adi_agarwal
New member
Demand Forecasting TODAY
New challenges, opportunities in e-business environment require practitioners to step beyond old boundaries and start marching into uncharted territory.
By Roger R. Gung, Ying Tat Leung, Grace Y. Lin and Roger Y. Tsai
________________________________________
Demand forecasting is a well-studied topic, and most techniques used in practice today are relatively mature. There are many commercial software packages for forecasting [12], ranging from stand-alone, specialized packages to modules in enterprise management systems. One would be inclined to conclude that there is no shortage of commercial software for demand forecasting. On the other hand, the now well-known paradigms of just-in-time manufacturing [9], time-based competition [2], quick response manufacturing [8], and more recently sense-and-respond [5] propose speed and responsiveness as a major competitive advantage, and hence a goal for manufacturing organizations. Under these frameworks, an organization can rely much less on a plan that is based on a demand forecast. One may then argue that demand forecasting no longer plays a critical role in the day-to-day operation of a business.
With a mature technology base, plenty of commercial software packages and a seemingly less critical role in an organization, is demand forecasting then a "solved problem"? Our experience has shown quite the opposite. We explore this issue by first discussing the business environment today, and then the new opportunities and challenges to demand forecasting presented by this new environment. As we shall see, the new opportunities and challenges are related to the abundance of information today and the intelligent choice and use of it. For each of several categories of information, we describe the nature of the data, the challenges in using them in forecasting, and offer some of our experiences whenever applicable. Finally, we emphasize that forecasting is only a means to an end. The revenue and profit perspective is important in setting priorities.
In this article, we discuss demand forecasting in the context of predicting future demand for products of a business organization. We focus on products of the physical type (such as electronic components, computers or automobiles), as opposed to services (such as long distance telephony or Web hosting). While demand forecasting for services exemplifies some of the issues discussed here, in other ways the service sector presents a different set of challenges suitable for another article.
The e-Business Environment
Despite the recent burst of the dot-com bubble, the new economy has indeed begun; it is an economy of fierce competition on a global basis. The consumer or any potential buyer has access to information any time of the day and more product choices than ever. "Word of mouth" on the Internet worldwide will increasingly become such a strong force in the market that any organization interested in surviving beyond a few weeks will have to present a clear and justifiable value proposition to the consumer. And if a business cannot deliver the value it promises, someone else will.
IBM coined the term "e-business" seven years ago. Today, 28 percent of global companies have substantially adopted e-business technology to derive a number of benefits — new revenue sources, better delivery of products, and significant cost savings. IBM has recently announced the next phase of adopting today's computing resources and global connectivity technologies to run an e-business: e-business on demand [1]. An on-demand e-business is embodied with end-to-end integrated processes across the entire company as well as with key partners, suppliers and customers such that the value chain as a whole can quickly respond to changes in demand, supply, customer preferences, market opportunity and competition.
Today, most consumers or buyers expect any respectable business to be an e-business, or at least to have the appearance of an e-business — that is, they can contact the business through electronic means, perform transactions with the business on the Web, and have their historical transaction record electronically available when they call. When a buyer attempts to purchase a product from a seller, he expects the product to be immediately available, or at the very least be told when it will be available with a high certainty. Otherwise he will most likely go to a competitor, which is usually one mouse click or phone call away. Such order fulfillment capability delivered on a consistent manner over time requires not only a lean and responsive manufacturing approach, but also serious effort in materials, capacity and logistics planning. Demand forecasting is an initial step for such planning and is therefore critical to the success of an e-business.
New Opportunities and Challenges in Demand Forecasting
From the viewpoint of a demand forecaster, the key opportunity presented by the new e-business environment is the abundance and availability of information, driven by the proliferation of information technology globally among businesses and consumers. To exploit such an opportunity, we need to be aware of the value of the different types of information and the subsequent exploration of it.
1. Information on demand throughput. The most well-established forecasting techniques are based on historical demand. In today's business environment, changes in the marketplace are swift and sudden, and may not follow the historical pattern; hence future demand may not be predicted accurately by relying on past demand alone.
First, we note that historical demand information need not be information about the past in the traditional sense, such as realized demand in the last month. Demand information about the present is commonly available. For instance, predicting demand in a time period when some customer orders are already placed can benefit from information on the incoming orders. Take for example a manufacturer serving other businesses. If the time period is a month, then a useful indicator might be the cumulative, month-to-date shipment quantity, quantity sold, or quantity sold to end-users.
Shipment refers to the quantity of shipment from a manufacturing site to a distribution center, owned either by the manufacturer or the customer. Quantity sold is the quantity of product shipped for which revenue has been received. Quantity sold to end-users is the quantity of products sold to the end consumers rather than the sales channels. Generally, end-user sales information becomes available to the manufacturer at a later time than that of the others, because it is usually provided by the sales channel — a middleman between the manufacturer and the end consumer. For IBM servers, end-user sales data availability generally lags behind the others by about a week. We found that for some geographic areas and product families, end-user sales are very useful for forecasting, while for others it can be erratic and unstable.
In some industries an initial "order" may not be a firm order; in this case the certainty of the order estimated subjectively or using an objective criterion will be useful information. A particular view of order information we have found useful for predicting demand is order quantity grouped by lead-time.
Second, some information, however small an amount, about demand in the near future may also be possible. In many industries, especially those who primarily serve other businesses, product sales are an important and often lengthy business process. It begins with the recognition of the need of a customer by a salesperson. It then proceeds from initial contact, through several stages of interaction between customer and supplier (including customer testing of some products such as large servers), to the creation and the signing of a contract. In general, when demand is strong, the speed and quantity of the flow of the sales process will have a corresponding indication. Such information is a longer range "headlight" than orders on hand, as it captures the flow of opportunities through the steps of the customer buying process.
In IBM, we refer to the identification of opportunities through their closure as sales as the "opportunity pipeline." We have found it to be very valuable in predicting aggregate demand or revenue, particularly in a time of change. In addition, we found it useful for predicting the success of sales for an individual customer or group of customers with similar attributes. This can be particularly useful for effective sales force resource allocation and future product design.
To take advantage of the information provided by these indicators, the data must be readily available. Clearly, such up-to-date order information would not be practical to obtain on a regular basis without fairly extensive use of information technology. Assuming that the raw data is available, data pre-processing and preparation is possibly the most time-consuming process among all of the forecasting challenges discussed in this article. For example, for IBM's own opportunity pipeline data, it requires a design of data-recording rules and/or processes so that sales personnel know what to record for a sales opportunity. The processes or rules need to be consistent, clear cut and easy to execute, and they need to produce adequate data for future forecasting. Furthermore, data collection during the execution of the sales process must be carried out consistently. This takes the cooperation of a wide range of functional units and different levels of management within the corporation. Also, forecasting efforts cannot start until there is sufficient history available.
2. Information on selling price and product promotion. Changes in the selling price and the presence of product promotions are known to have a significant effect on demand in many industries. Today, in large part due to the proliferation of information and other technologies, price changes are less costly. Price changes in electronic business-to-business product catalogs or Web-based retail businesses incur little incremental cost. Even in traditional retail stores, the day will soon come when a button on a computer is pressed to issue a price change, and new prices will be reflected on a liquid crystal shelf label in a physical store a few seconds later. Such opportunities imply that price changes and promotion actions may be used very frequently, and so they can no longer be analyzed separately from "normal" demand.
Product promotions are getting very sophisticated. Targeted marketing, and ultimately one-on-one marketing, has created complications in the analysis of promotion effects. The traditional way of applying a general "lift factor" to nominal demand when a certain promotion is performed may not be adequate. At the very least, this "lift factor" needs to take into account the promotion's target portion of the entire market, a quantity to be estimated itself.
To enhance forecast accuracy with pricing and promotion information, a prerequisite is a well-maintained and managed database of price and promotions corresponding to historical demand information. This requires a disciplined process to capture the information in a timely manner. Even more challenging is collecting and maintaining historical data of competitors' promotions and price changes. Such information is especially crucial in industries where products are of the commodity type. It is generally easier to collect such information for retail businesses. For traditional brick-and-mortar stores, organizations have long used third-party vendors to do "competitive shopping" (i.e., to collect product prices of competitors' stores). To collect price information from Web-based retailers, crawler software is available [7].
Although it is well publicized [6, 10] that good price and promotion management improves revenue, it is interesting to note that it is generally not clear whether the improvement comes from an accurate forecast of price or promotion effects, or if it is the result of an optimization based on fairly rough forecasts.
3. Information on product life cycle. One of the serious challenges facing a demand forecaster in the e-business environment is ever shortening product life cycles. In many industries, a product can be expected to have a life of at most one year. As is customary, it can inherit older history from its predecessor product, which can in turn inherit history from its own predecessor and so on. This means that in order to get, say, two to three years history, we need a well-organized product map over time. At this point in time, we have found that many organizations do not have such product map data stored in a usable manner. The upcoming industry of product life cycle management software will no doubt provide a better infrastructure to maintain such information. However, even with a product map, one would not go too far back since the entire business environment was different. For many products, we have practically at most two to three years of appropriate history.
For ease of product management, most, if not all, organizations use a product hierarchy. The hierarchy is organized by a set of attributes of the products, such as marketing attributes (e.g. geographical area, customer type) or technical attributes of the products (e.g. speed or other capacity). Because individual products are phased out and new products come in constantly over time, and the fact that the hierarchy might be reorganized fairly frequently to reflect the fast changing business environment (e.g. gaining or losing major customers or markets), the product hierarchy is dynamic.
With a sizable number of products in its portfolio, organizations now have a different product hierarchy every month. This presents at least two challenges. First, we need to find a way to obtain the history of every node in this changing hierarchy. For nodes representing individual products, we can use inheritance as described. For nodes at levels higher than the individual products, we need to "reconstruct" its history every time the hierarchy is used, based on the current children of the node. Note, the history of each child can be a constructed history of several generations. This results in a complex data preprocessing step. Further, it is arguable that such construction may not necessarily be the best or even correct way of handling a dynamic product hierarchy. Second, the historical forecast made in the past no longer corresponds to the newly constructed history or to the most current product hierarchy. It is not clear how we can obtain running statistics of historical forecast errors. Subsequently, forecast monitoring becomes difficult.
In planning product transition, we are interested in forecasting the demand of a single product, not the sum of itself and its predecessor. In this case it is natural to try using the current stage of a product's life cycle to help its demand prediction. Existing state of the art in applying life cycle information to forecasting has not been fully proven and demonstrated in a real business environment, partially due to a lack of organized data mentioned before and a lack of serious effort in applying it in full scale. Fortunately, when forecasting demand of aggregated product groups in which the rise and fall in demand within the life cycle of an individual product compensates for each other, it is less critical to consider life cycle information.
4. Information on the marketplace. As econometricians have long known, demand history is only one of many streams of information from which a forecast can be made. The e-business environment presents at least two key opportunities on forecast information. First, high level indicators of economic activities such as total production output of an industry are more up-to-date than previously possible. Data are collected continuously and automatically in electronic transactions, and should also be less error prone. This will be increasingly so as more and more business-to-business as well as business-to-consumer transactions are performed electronically. Note that this comment applies to retail store transactions as well, where the transaction is performed electronically at the point-of-sale and will be recorded into some central database. Second, more detailed economic data are available, such as those by product types within an industry. Our experience shows that detailed data are more useful as a predictor of the demand of a single product of an individual organization.
5. Information on consumers. We have already mentioned the use of end user sales or consumer demand as a source of demand information. Demand history consisting of the quantity sold, and perhaps selling price, is no longer the only piece(s) of information coming directly from past consumers. Customer database collected by an organization over time, previously limited only to expensive products such as mainframe computers or automobiles, is now likely to cover regular products such as end-user software packages or even children's products. Curiously, for these two product types, end users have very different incentives to register with the manufacturer: for future product updates in the form of software downloads, or product recalls. As more products incorporate elements of software that will go through a typical life cycle of updates, customers are more likely to register with the product manufacturer. Thus the existing customer base is no longer characterized only by a total sales number, but rather a database of information at the manufacturer's own request.
Customer loyalty programs, heralded by frequent-flyer programs offered by airlines, identify the customer, who can be anonymous, whenever a product is sold. This enables an organization to have a view of a single customer over time. One can now go to a grocery store's Web site, enter one's loyalty card number, and retrieve a record of all one has purchased from that store for the last 12 months. It would seem that such a trend of having centralized information about customers over time will only continue. For example, with the proliferation of consumer credit cards, the purchase history of a household across many product categories is available in the transaction database of the credit card issuer.
Perhaps even more importantly, the e-business environment provides an opportunity to readily collect information about potential or future buyers. Along the different channels of communication between an organization and its potential customers, most companies now routinely log every visit to the product Web pages, every call made to an inquiry response center and every e-mail that was received. Many organizations also use every customer touch point as an opportunity to perform a brief customer survey to collect information about their customers and comments on their products.
The challenge here, of course, is making use of such information to predict future demand more reliably.
A Means to an End: The Profit and Revenue Perspective
Although forecast accuracy is one of the most important targets driving this forecasting enhancement effort, one should not lose sight of the fact that one of the most important purposes of doing forecasting is to improve the efficiency of the business including the bottom-line profit. Forecasting is only a means to an end. Understanding what roles forecasting plays in the process of optimizing profit would set the right perspective for the forecaster and enable him to place the right resource in the right area, and to set the appropriate goal. Our own work in this area of relating forecasting to profit optimization [11] has generated some interesting observations. For example, for products with a high-profit margin, the decisions on product supply and inventory to optimize profit is not very sensitive to the forecast accuracy, when compared to products with a low-profit margin. We also found that the "process window" for the product supply to vary, while maintaining near-maximum profit, increases as the profit margin increases. This process window gives a window of opportunity for the supply decision-maker to take other factors into consideration, such as customer satisfaction, financial objectives and other business constraints. Furthermore, the wider the window, the more room it leaves for forecast error. That is, the tolerance or target for forecast accuracy depends on the profit margin and the corresponding width of the process window. It is obvious from this discussion that for a high-margin product, the forecast can be fairly inaccurate while the product can remain highly profitable. But for a product with a low-profit margin, unless the forecast accuracy is very high, it is hard to remain profitable.
Another observation points to the importance of forecasting the bias and variation of the demand rather than just forecasting an average demand. In fact, not only the mean and standard deviation of the demand needs to be estimated, but the uncertainty of the mean and standard deviation estimate plays a role in the supply optimization. The uncertainty of the demand estimate helps pinpoint which portion of the process window is reliable. This analysis also quantifies the effect of forecast uncertainty on the profit, giving some guidance to where and how resources for forecasting should be placed to maximize profit. Without a perspective of profit versus supply given the demand uncertainty, it would be difficult for the forecaster to know what is important to measure, what metric is needed, and what performance target to shoot for.
Conclusion
We have offered our view of the new challenges and opportunities for demand forecasting in the e-business environment today. Among other works, we have developed some enhanced forecasting tools, such as a proprietary forecasting tool (BIA), Forecast Improvement Tool (FIT), forecasting using customer surveys [4] and diffusion-based forecasting model to predict product demand over its entire life [3]. These tools take advantage of some of the information mentioned above. We have applied these tools to some of our diverse businesses and have experienced certain success. We have also developed a methodology called ProfitMax [11] for supply decision-making in several major lines of hardware products and have found improvement on our bottom line. Our experience with these applications form the basis for our views and observations here.
We recommend that forecasting practitioners step beyond the old boundaries and start marching into this new territory. Look for all those indicators mentioned above in your company and initiate effort to collect, store and maintain them. Start thinking of ways to relate this information to future demand, but do not limit yourselves to traditional time-series methods or linear regression models. Find ways to strengthen the forecast using more indicators, but do not limit yourself to a linearly-weighted combination of the multiple forecasts. There are ample opportunities to develop your own methodologies and apply them to your business to gain the value of new information now available. Forecasting may never be the same.
References
1. "e-Business on Demand," http://www.ibm.com/ondemand
2. J.D. Blackburn, "Time-Based-Competition: The Next Battleground in American Manufacturing," Irwin Professional Publication, 1990.
3. R.R. Gung, Y. Jang, G. Lin, R. Tsai, "A Bass Diffusion-based Product Lifecycle and Demand Forecasting Model," IBM Research Report, November 2002.
4. A. Heching, Y.T. Leung, J. Caruso, "Using Surveys to Understand the Present and Predict the Future," in Proceedings of the 14th Annual Conference of the Northeast SAS Users' Group, September 2001.
5. G. Lin et al., "The Sense and Response Enterprise," OR/MS Today, Vol. 29, No. 2, April 2002.
6. G. McWilliams, "Dell Fine-Tunes Its PC Pricing to Gain an Edge in Slow Market," The Wall Street Journal, June 8, 2001.
7. R.C. Miller and K. Bharat, "SPHINX: A Framework for Creating Personal, Site-Specific Web Crawlers," in Proceedings of WWW7, Brisbane, Australia, April 1998.
8. R. Suri, "Quick Response Manufacturing: A Competitive Strategy for the 21st Century," Proceedings of the Quick Response Manufacturing Conference, June 2000, Dearborn, Mich.: Society of Manufacturing Engineers.
9. C.A. Swanson, W.M. Lankford, "JIT Manufacturing," Business Process Management Journal, Vol. 4, No. 4, 1998, pp. 333-341.
10. B. Tedechi, "Scientifically Priced Retail Goods," The New York Times, Sept. 2, 2002.
11. R. Tsai, M. Ettl, Y. Lee, J. Konopka, S. Liu, F. Cheng, "ProfitMax, Risk Management and Supply Decision Support: Theory and Methodology," IBM Research Report, November 2002.
12. J. Yurkiewicz, "Software Survey: Forecasting 2000," OR/MS Today, Vol. 27, No. 1, February 2000.
New challenges, opportunities in e-business environment require practitioners to step beyond old boundaries and start marching into uncharted territory.
By Roger R. Gung, Ying Tat Leung, Grace Y. Lin and Roger Y. Tsai
________________________________________
Demand forecasting is a well-studied topic, and most techniques used in practice today are relatively mature. There are many commercial software packages for forecasting [12], ranging from stand-alone, specialized packages to modules in enterprise management systems. One would be inclined to conclude that there is no shortage of commercial software for demand forecasting. On the other hand, the now well-known paradigms of just-in-time manufacturing [9], time-based competition [2], quick response manufacturing [8], and more recently sense-and-respond [5] propose speed and responsiveness as a major competitive advantage, and hence a goal for manufacturing organizations. Under these frameworks, an organization can rely much less on a plan that is based on a demand forecast. One may then argue that demand forecasting no longer plays a critical role in the day-to-day operation of a business.
With a mature technology base, plenty of commercial software packages and a seemingly less critical role in an organization, is demand forecasting then a "solved problem"? Our experience has shown quite the opposite. We explore this issue by first discussing the business environment today, and then the new opportunities and challenges to demand forecasting presented by this new environment. As we shall see, the new opportunities and challenges are related to the abundance of information today and the intelligent choice and use of it. For each of several categories of information, we describe the nature of the data, the challenges in using them in forecasting, and offer some of our experiences whenever applicable. Finally, we emphasize that forecasting is only a means to an end. The revenue and profit perspective is important in setting priorities.
In this article, we discuss demand forecasting in the context of predicting future demand for products of a business organization. We focus on products of the physical type (such as electronic components, computers or automobiles), as opposed to services (such as long distance telephony or Web hosting). While demand forecasting for services exemplifies some of the issues discussed here, in other ways the service sector presents a different set of challenges suitable for another article.
The e-Business Environment
Despite the recent burst of the dot-com bubble, the new economy has indeed begun; it is an economy of fierce competition on a global basis. The consumer or any potential buyer has access to information any time of the day and more product choices than ever. "Word of mouth" on the Internet worldwide will increasingly become such a strong force in the market that any organization interested in surviving beyond a few weeks will have to present a clear and justifiable value proposition to the consumer. And if a business cannot deliver the value it promises, someone else will.
IBM coined the term "e-business" seven years ago. Today, 28 percent of global companies have substantially adopted e-business technology to derive a number of benefits — new revenue sources, better delivery of products, and significant cost savings. IBM has recently announced the next phase of adopting today's computing resources and global connectivity technologies to run an e-business: e-business on demand [1]. An on-demand e-business is embodied with end-to-end integrated processes across the entire company as well as with key partners, suppliers and customers such that the value chain as a whole can quickly respond to changes in demand, supply, customer preferences, market opportunity and competition.
Today, most consumers or buyers expect any respectable business to be an e-business, or at least to have the appearance of an e-business — that is, they can contact the business through electronic means, perform transactions with the business on the Web, and have their historical transaction record electronically available when they call. When a buyer attempts to purchase a product from a seller, he expects the product to be immediately available, or at the very least be told when it will be available with a high certainty. Otherwise he will most likely go to a competitor, which is usually one mouse click or phone call away. Such order fulfillment capability delivered on a consistent manner over time requires not only a lean and responsive manufacturing approach, but also serious effort in materials, capacity and logistics planning. Demand forecasting is an initial step for such planning and is therefore critical to the success of an e-business.
New Opportunities and Challenges in Demand Forecasting
From the viewpoint of a demand forecaster, the key opportunity presented by the new e-business environment is the abundance and availability of information, driven by the proliferation of information technology globally among businesses and consumers. To exploit such an opportunity, we need to be aware of the value of the different types of information and the subsequent exploration of it.
1. Information on demand throughput. The most well-established forecasting techniques are based on historical demand. In today's business environment, changes in the marketplace are swift and sudden, and may not follow the historical pattern; hence future demand may not be predicted accurately by relying on past demand alone.
First, we note that historical demand information need not be information about the past in the traditional sense, such as realized demand in the last month. Demand information about the present is commonly available. For instance, predicting demand in a time period when some customer orders are already placed can benefit from information on the incoming orders. Take for example a manufacturer serving other businesses. If the time period is a month, then a useful indicator might be the cumulative, month-to-date shipment quantity, quantity sold, or quantity sold to end-users.
Shipment refers to the quantity of shipment from a manufacturing site to a distribution center, owned either by the manufacturer or the customer. Quantity sold is the quantity of product shipped for which revenue has been received. Quantity sold to end-users is the quantity of products sold to the end consumers rather than the sales channels. Generally, end-user sales information becomes available to the manufacturer at a later time than that of the others, because it is usually provided by the sales channel — a middleman between the manufacturer and the end consumer. For IBM servers, end-user sales data availability generally lags behind the others by about a week. We found that for some geographic areas and product families, end-user sales are very useful for forecasting, while for others it can be erratic and unstable.
In some industries an initial "order" may not be a firm order; in this case the certainty of the order estimated subjectively or using an objective criterion will be useful information. A particular view of order information we have found useful for predicting demand is order quantity grouped by lead-time.
Second, some information, however small an amount, about demand in the near future may also be possible. In many industries, especially those who primarily serve other businesses, product sales are an important and often lengthy business process. It begins with the recognition of the need of a customer by a salesperson. It then proceeds from initial contact, through several stages of interaction between customer and supplier (including customer testing of some products such as large servers), to the creation and the signing of a contract. In general, when demand is strong, the speed and quantity of the flow of the sales process will have a corresponding indication. Such information is a longer range "headlight" than orders on hand, as it captures the flow of opportunities through the steps of the customer buying process.
In IBM, we refer to the identification of opportunities through their closure as sales as the "opportunity pipeline." We have found it to be very valuable in predicting aggregate demand or revenue, particularly in a time of change. In addition, we found it useful for predicting the success of sales for an individual customer or group of customers with similar attributes. This can be particularly useful for effective sales force resource allocation and future product design.
To take advantage of the information provided by these indicators, the data must be readily available. Clearly, such up-to-date order information would not be practical to obtain on a regular basis without fairly extensive use of information technology. Assuming that the raw data is available, data pre-processing and preparation is possibly the most time-consuming process among all of the forecasting challenges discussed in this article. For example, for IBM's own opportunity pipeline data, it requires a design of data-recording rules and/or processes so that sales personnel know what to record for a sales opportunity. The processes or rules need to be consistent, clear cut and easy to execute, and they need to produce adequate data for future forecasting. Furthermore, data collection during the execution of the sales process must be carried out consistently. This takes the cooperation of a wide range of functional units and different levels of management within the corporation. Also, forecasting efforts cannot start until there is sufficient history available.
2. Information on selling price and product promotion. Changes in the selling price and the presence of product promotions are known to have a significant effect on demand in many industries. Today, in large part due to the proliferation of information and other technologies, price changes are less costly. Price changes in electronic business-to-business product catalogs or Web-based retail businesses incur little incremental cost. Even in traditional retail stores, the day will soon come when a button on a computer is pressed to issue a price change, and new prices will be reflected on a liquid crystal shelf label in a physical store a few seconds later. Such opportunities imply that price changes and promotion actions may be used very frequently, and so they can no longer be analyzed separately from "normal" demand.
Product promotions are getting very sophisticated. Targeted marketing, and ultimately one-on-one marketing, has created complications in the analysis of promotion effects. The traditional way of applying a general "lift factor" to nominal demand when a certain promotion is performed may not be adequate. At the very least, this "lift factor" needs to take into account the promotion's target portion of the entire market, a quantity to be estimated itself.
To enhance forecast accuracy with pricing and promotion information, a prerequisite is a well-maintained and managed database of price and promotions corresponding to historical demand information. This requires a disciplined process to capture the information in a timely manner. Even more challenging is collecting and maintaining historical data of competitors' promotions and price changes. Such information is especially crucial in industries where products are of the commodity type. It is generally easier to collect such information for retail businesses. For traditional brick-and-mortar stores, organizations have long used third-party vendors to do "competitive shopping" (i.e., to collect product prices of competitors' stores). To collect price information from Web-based retailers, crawler software is available [7].
Although it is well publicized [6, 10] that good price and promotion management improves revenue, it is interesting to note that it is generally not clear whether the improvement comes from an accurate forecast of price or promotion effects, or if it is the result of an optimization based on fairly rough forecasts.
3. Information on product life cycle. One of the serious challenges facing a demand forecaster in the e-business environment is ever shortening product life cycles. In many industries, a product can be expected to have a life of at most one year. As is customary, it can inherit older history from its predecessor product, which can in turn inherit history from its own predecessor and so on. This means that in order to get, say, two to three years history, we need a well-organized product map over time. At this point in time, we have found that many organizations do not have such product map data stored in a usable manner. The upcoming industry of product life cycle management software will no doubt provide a better infrastructure to maintain such information. However, even with a product map, one would not go too far back since the entire business environment was different. For many products, we have practically at most two to three years of appropriate history.
For ease of product management, most, if not all, organizations use a product hierarchy. The hierarchy is organized by a set of attributes of the products, such as marketing attributes (e.g. geographical area, customer type) or technical attributes of the products (e.g. speed or other capacity). Because individual products are phased out and new products come in constantly over time, and the fact that the hierarchy might be reorganized fairly frequently to reflect the fast changing business environment (e.g. gaining or losing major customers or markets), the product hierarchy is dynamic.
With a sizable number of products in its portfolio, organizations now have a different product hierarchy every month. This presents at least two challenges. First, we need to find a way to obtain the history of every node in this changing hierarchy. For nodes representing individual products, we can use inheritance as described. For nodes at levels higher than the individual products, we need to "reconstruct" its history every time the hierarchy is used, based on the current children of the node. Note, the history of each child can be a constructed history of several generations. This results in a complex data preprocessing step. Further, it is arguable that such construction may not necessarily be the best or even correct way of handling a dynamic product hierarchy. Second, the historical forecast made in the past no longer corresponds to the newly constructed history or to the most current product hierarchy. It is not clear how we can obtain running statistics of historical forecast errors. Subsequently, forecast monitoring becomes difficult.
In planning product transition, we are interested in forecasting the demand of a single product, not the sum of itself and its predecessor. In this case it is natural to try using the current stage of a product's life cycle to help its demand prediction. Existing state of the art in applying life cycle information to forecasting has not been fully proven and demonstrated in a real business environment, partially due to a lack of organized data mentioned before and a lack of serious effort in applying it in full scale. Fortunately, when forecasting demand of aggregated product groups in which the rise and fall in demand within the life cycle of an individual product compensates for each other, it is less critical to consider life cycle information.
4. Information on the marketplace. As econometricians have long known, demand history is only one of many streams of information from which a forecast can be made. The e-business environment presents at least two key opportunities on forecast information. First, high level indicators of economic activities such as total production output of an industry are more up-to-date than previously possible. Data are collected continuously and automatically in electronic transactions, and should also be less error prone. This will be increasingly so as more and more business-to-business as well as business-to-consumer transactions are performed electronically. Note that this comment applies to retail store transactions as well, where the transaction is performed electronically at the point-of-sale and will be recorded into some central database. Second, more detailed economic data are available, such as those by product types within an industry. Our experience shows that detailed data are more useful as a predictor of the demand of a single product of an individual organization.
5. Information on consumers. We have already mentioned the use of end user sales or consumer demand as a source of demand information. Demand history consisting of the quantity sold, and perhaps selling price, is no longer the only piece(s) of information coming directly from past consumers. Customer database collected by an organization over time, previously limited only to expensive products such as mainframe computers or automobiles, is now likely to cover regular products such as end-user software packages or even children's products. Curiously, for these two product types, end users have very different incentives to register with the manufacturer: for future product updates in the form of software downloads, or product recalls. As more products incorporate elements of software that will go through a typical life cycle of updates, customers are more likely to register with the product manufacturer. Thus the existing customer base is no longer characterized only by a total sales number, but rather a database of information at the manufacturer's own request.
Customer loyalty programs, heralded by frequent-flyer programs offered by airlines, identify the customer, who can be anonymous, whenever a product is sold. This enables an organization to have a view of a single customer over time. One can now go to a grocery store's Web site, enter one's loyalty card number, and retrieve a record of all one has purchased from that store for the last 12 months. It would seem that such a trend of having centralized information about customers over time will only continue. For example, with the proliferation of consumer credit cards, the purchase history of a household across many product categories is available in the transaction database of the credit card issuer.
Perhaps even more importantly, the e-business environment provides an opportunity to readily collect information about potential or future buyers. Along the different channels of communication between an organization and its potential customers, most companies now routinely log every visit to the product Web pages, every call made to an inquiry response center and every e-mail that was received. Many organizations also use every customer touch point as an opportunity to perform a brief customer survey to collect information about their customers and comments on their products.
The challenge here, of course, is making use of such information to predict future demand more reliably.
A Means to an End: The Profit and Revenue Perspective
Although forecast accuracy is one of the most important targets driving this forecasting enhancement effort, one should not lose sight of the fact that one of the most important purposes of doing forecasting is to improve the efficiency of the business including the bottom-line profit. Forecasting is only a means to an end. Understanding what roles forecasting plays in the process of optimizing profit would set the right perspective for the forecaster and enable him to place the right resource in the right area, and to set the appropriate goal. Our own work in this area of relating forecasting to profit optimization [11] has generated some interesting observations. For example, for products with a high-profit margin, the decisions on product supply and inventory to optimize profit is not very sensitive to the forecast accuracy, when compared to products with a low-profit margin. We also found that the "process window" for the product supply to vary, while maintaining near-maximum profit, increases as the profit margin increases. This process window gives a window of opportunity for the supply decision-maker to take other factors into consideration, such as customer satisfaction, financial objectives and other business constraints. Furthermore, the wider the window, the more room it leaves for forecast error. That is, the tolerance or target for forecast accuracy depends on the profit margin and the corresponding width of the process window. It is obvious from this discussion that for a high-margin product, the forecast can be fairly inaccurate while the product can remain highly profitable. But for a product with a low-profit margin, unless the forecast accuracy is very high, it is hard to remain profitable.
Another observation points to the importance of forecasting the bias and variation of the demand rather than just forecasting an average demand. In fact, not only the mean and standard deviation of the demand needs to be estimated, but the uncertainty of the mean and standard deviation estimate plays a role in the supply optimization. The uncertainty of the demand estimate helps pinpoint which portion of the process window is reliable. This analysis also quantifies the effect of forecast uncertainty on the profit, giving some guidance to where and how resources for forecasting should be placed to maximize profit. Without a perspective of profit versus supply given the demand uncertainty, it would be difficult for the forecaster to know what is important to measure, what metric is needed, and what performance target to shoot for.
Conclusion
We have offered our view of the new challenges and opportunities for demand forecasting in the e-business environment today. Among other works, we have developed some enhanced forecasting tools, such as a proprietary forecasting tool (BIA), Forecast Improvement Tool (FIT), forecasting using customer surveys [4] and diffusion-based forecasting model to predict product demand over its entire life [3]. These tools take advantage of some of the information mentioned above. We have applied these tools to some of our diverse businesses and have experienced certain success. We have also developed a methodology called ProfitMax [11] for supply decision-making in several major lines of hardware products and have found improvement on our bottom line. Our experience with these applications form the basis for our views and observations here.
We recommend that forecasting practitioners step beyond the old boundaries and start marching into this new territory. Look for all those indicators mentioned above in your company and initiate effort to collect, store and maintain them. Start thinking of ways to relate this information to future demand, but do not limit yourselves to traditional time-series methods or linear regression models. Find ways to strengthen the forecast using more indicators, but do not limit yourself to a linearly-weighted combination of the multiple forecasts. There are ample opportunities to develop your own methodologies and apply them to your business to gain the value of new information now available. Forecasting may never be the same.
References
1. "e-Business on Demand," http://www.ibm.com/ondemand
2. J.D. Blackburn, "Time-Based-Competition: The Next Battleground in American Manufacturing," Irwin Professional Publication, 1990.
3. R.R. Gung, Y. Jang, G. Lin, R. Tsai, "A Bass Diffusion-based Product Lifecycle and Demand Forecasting Model," IBM Research Report, November 2002.
4. A. Heching, Y.T. Leung, J. Caruso, "Using Surveys to Understand the Present and Predict the Future," in Proceedings of the 14th Annual Conference of the Northeast SAS Users' Group, September 2001.
5. G. Lin et al., "The Sense and Response Enterprise," OR/MS Today, Vol. 29, No. 2, April 2002.
6. G. McWilliams, "Dell Fine-Tunes Its PC Pricing to Gain an Edge in Slow Market," The Wall Street Journal, June 8, 2001.
7. R.C. Miller and K. Bharat, "SPHINX: A Framework for Creating Personal, Site-Specific Web Crawlers," in Proceedings of WWW7, Brisbane, Australia, April 1998.
8. R. Suri, "Quick Response Manufacturing: A Competitive Strategy for the 21st Century," Proceedings of the Quick Response Manufacturing Conference, June 2000, Dearborn, Mich.: Society of Manufacturing Engineers.
9. C.A. Swanson, W.M. Lankford, "JIT Manufacturing," Business Process Management Journal, Vol. 4, No. 4, 1998, pp. 333-341.
10. B. Tedechi, "Scientifically Priced Retail Goods," The New York Times, Sept. 2, 2002.
11. R. Tsai, M. Ettl, Y. Lee, J. Konopka, S. Liu, F. Cheng, "ProfitMax, Risk Management and Supply Decision Support: Theory and Methodology," IBM Research Report, November 2002.
12. J. Yurkiewicz, "Software Survey: Forecasting 2000," OR/MS Today, Vol. 27, No. 1, February 2000.
