Data mining in the real world: What do we need and what do we have?

Historically, data mining has been in the hands of small teams of expert statisticians who produce a few models per year. However, recently companies invested heavily in building huge data warehouses (from a few terabytes to peta-bytes) that contain millions of records and thousands of variables; for example, 5,000 variables on 150 million customers and prospects. That has changed the economics of data mining. Now businesses want a return on that investment and are looking well beyond reporting and basic statistics [1]. When they review their business activities, they see the need for 100s or 1000s of predictive models per year. Of course, very few companies can produce that many today, due to a lack of expert staff and appropriate tools. Some actually do generate that many models and we will provide examples, including: • A broadband communications company moved from 5 cross-sell models per year to 1600 • A wireless communications company that produces 700 CRM models per year • A national retailers cut time-to-model by 90 % and scores 75M households in 30 minutes • A marketing research firm built 370 propensity-to-buy models on a PC in an afternoon We will analyze model production from an industrial viewpoint and review constraints such as large data volumes (records and variables), ability to produce robust models with littl