Tag Archives: data

Controlled Experiments for Software Solutions

Jeff Fry linked to a great webcast in Controlled Experiments To Test For Bugs In Our Mental Models.

I firmly believe that applied statistics-based experiments are under-appreciated by businesses (and, for that matter, business schools). Few people who understand them are as articulate and concise as Kohavi. Admittedly, I could be accused of being biased as: (a) I am the son of a prominent applied statistician and (b) I am the founder of a software testing tools company that uses applied statistics-based methods and algorithms to make our tool work.

Summary of the webcast, on Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO – a presentation by Ron Kohavi with Microsoft Research.

1:00 Amazon: in 2000, Greg Linden wanted to add recommendations in shopping cards during the check out process. The “HiPPO” (meaning the Highest Paid Person’s Opinion) was against it on the grounds that it would be a bad idea; recommendations would confuse and/or distract people. Amazon, a company with a good culture of experimentation, decided to run a small experiment anyway, “just to get the data” – It was wildly successful and is in widespread use today at Amazon and other firms.

3:00 Dr. Footcare example: Including a coupon code above the total price to be paid had a dramatic impact on abandonment rates.

4:00 “Was this answer useful?” Dramatic differences occur when Y/N is replaced with 5 Stars and whether an empty text box is initially shown with either (or whether it is triggered only after a user clicks to give their initial response)

6:00 Sewing machines: experimenting with a sales promotion strategy led to extremely counter-intuitive pricing choice

7:00 “We are really, really bad at understanding what is going to work with customers…”
Continue reading →

Data Analysts Captivated by R’s Power

data mining has entered a golden age, whether being used to set ad prices, find new drugs more quickly or fine-tune financial models. Companies as diverse as Google, Pfizer, Merck, Bank of America, the InterContinental Hotels Group and Shell use it.
…
Close to 1,600 different packages reside on just one of the many Web sites devoted to R, and the number of packages has grown exponentially. One package, called BiodiversityR, offers a graphical interface aimed at making calculations of environmental trends easier.

Another package, called Emu, analyzes speech patterns, while GenABEL is used to study the human genome. The financial services community has demonstrated a particular affinity for R; dozens of packages exist for derivatives analysis alone. “The great beauty of R is that you can modify it to do all sorts of things,” said Hal Varian, chief economist at Google. “And you have a lot of prepackaged stuff that’s already available, so you’re standing on the shoulders of giants.”

R first appeared in 1996, when the statistics professors Ross Ihaka and Robert Gentleman of the University of Auckland in New Zealand released the code as a free software package. According to them, the notion of devising something like R sprang up during a hallway conversation. They both wanted technology better suited for their statistics students, who needed to analyze data and produce graphical models of the information. Most comparable software had been designed by computer scientists and proved hard to use.

R is another example of great, free, open source software. See R packages for Statistics for Experimenters.

via: R in the news

Correlation is Not Causation: “Fat is Catching” Theory Exposed

“Fat is catching” theory exposed

Their study was reported to have shown that you can “catch” obesity from having fat friends and that obesity is so contagious, it can be spread long-distance by email and instant messaging. Even healthcare professionals, who didn’t understand the etiology of true obesity or how statistics can be misused, failed to detect the implausibility of “second-hand obesity.” In fact, some doctors became so enamored with the new “science of networking” they believed it should be a new medical specialty: network medicine.
…
Jason M. Fletcher, Ph.D., assistant professor at the Yale School of Public Health in New Haven, Connecticut, along with Boston economist, Ethan Cohen-Cole, Ph.D., designed an ingenious study. They selected conditions that no one would seriously believe were spread by social networking and online friendships: height, headaches and acne. They then applied the same standard statistical methods used in Christakis and Fowler’s social networking research to “find” that acne, height and headaches have the same “social network effect.”
…
As they explained, patterns of association among people can lead to correlations in health conditions between friends that are not caused by direct social network effects at all.
…
There is a need for caution when attributing causality to correlations in health outcomes between friends using non-experimental data. Confounding is only one of many empirical challenges to estimating social network effects.

Excellent reminder of the risks of analyzing data for correlations. We continue to, far to often, fail to interpret data properly. Both authors of the study, received PhD’s from the University of Wisconsin-Madison which strengthens my belief that it is teaching students well (just kidding).

Also another example of the scientific inquiry process where scientists challenge the conclusions drawn by other scientists. It is a wonderful system, even if confusing and not the clean idea so many have of how science works.

Poor Reporting and Unfounded Implications

Correlation is not causation. And reporting of the form, “1 time this happened” and so I report it as though it is some relevant fact, is sad. Take any incident that happened and then state random traits you want to imply there is some relevant link to (blue eyes, red hair, people that watch IT Crowd, people that bought a banana yesterday, tall, overweight, did poorly in math…) and most people will know you are ignorant.

Looking at random data people will find patterns. Sound scientific experimentation is how we learn, not trying to find anything that support our opinions. Statistics don’t lie but ignorant people draw faulty conclusions from data (when they are innumerate – illiteracy with mathematical concepts).

It’s not what the papers say, it’s what they don’t by Ben Goldacre

On Tuesday the Telegraph, the Independent, the Mirror, the Express, the Mail, and the Metro all reported that a coroner was hearing the case of a toddler who died after receiving the MMR vaccine, which the parents blamed for their loss. Toddler ‘died after MMR jab’ (Metro), ‘Healthy’ baby died after MMR jab (Independent), you know the headlines by now.

On Thursday the coroner announced his verdict: the vaccine played no part in this child’s death. So far, of the papers above, only the Telegraph has had the decency to cover the outcome.
…
Measles cases are rising. Middle class parents are not to blame, even if they do lack rhetorical panache when you try to have a discussion with them about it.

They have been systematically and vigorously misled by the media, the people with access to all the information, who still choose, collectively, between themselves, so robustly that it might almost be a conspiracy, to give you only half the facts.

Science education is important. Even if people do not become scientists, ignorance of scientific thinking is dangerous. The lack of scientific literacy allows scientifically illiterate leaders to make claims that are lacking scientific merit. And results in people making poor choices themselves, due to their ignorance.

Compounding is the Most Powerful Force in the Universe

A talking head with some valuable info. I remember my father (a statistics professor) getting me to understand this as a small child (about 6 years old). The concept of growth and mathematical compounding is an important idea to understand as you think and learn about the world. It also is helpful so you understand that statistics don’t lie but ignorant people can draw false conclusions from limited data.

It is unclear if Einstein really said this but he is often quoted as saying “compounding is the most powerful force in the universe.” Whether he did or not, understanding this simple concept is a critical component of numeracy (literacy with numbers). Also quoted at times as: “Compound interest is the eighth wonder of the world.” My guess is that people just find the concept of compounding amazing and then attribute quotes about it to Einstein.

I strongly encourage you to watch at least the first 2 segments (a total of 15 minutes). And then take some time and think. Take some time to think about compounding in ways to help you internalize the concepts. You can also read his book: The Essential Exponential For the Future of Our Planet by Albert Bartlett.

How Large Quantities of Information Change Everything

Scale: How Large Quantities of Information Change Everything

There’s another important downside to scale. When we look at large quantities of information, what we’re really doing is searching for patterns. And being the kind of creatures that we are, and given the nature of the laws of probability, we are going to find patterns. Distinguishing between a real legitimate pattern, and something random that just happens to look like a pattern can be somewhere between difficult and impossible. Using things like Bayesian methods to screen out the false positives can help, but scale means that scientists need to learn new methods – both the new ways of doing things that they couldn’t do before, and the new ways of recognizing when they’ve screwed up.

There’s the nature of scale. Tasks that were once simple have become hard or even impossible, because they can’t be done at scale. Tasks that were once impossible have become easy because scale makes them possible. Scale changes everything.

I discussed related ideas on my Curious Cat Management Improvement blog recently: Does the Data Deluge Make the Scientific Method Obsolete?

Rate of Cancer Detected and Death Rates Declines

Declines in Cancer Incidence and Death Rates in report from the National Cancer Institute and CDC:

“The drop in incidence seen in this year’s Annual Report is something we’ve been waiting to see for a long time,” said Otis W. Brawley, M.D., chief medical officer of the American Cancer Society (ACS). “However, we have to be somewhat cautious about how we interpret it, because changes in incidence can be caused not only by reductions in risk factors for cancer, but also by changes in screening practices. Regardless, the continuing drop in mortality is evidence once again of real progress made against cancer, reflecting real gains in prevention, early detection, and treatment.”
…
According to a U.S. Surgeon General’s report, cigarette smoking accounts for approximately 30 percent of all cancer deaths, with lung cancer accounting for 80 percent of the smoking-attributable cancer deaths. Other cancers caused by smoking include cancers of the oral cavity, pharynx, larynx, esophagus, stomach, bladder, pancreas, liver, kidney, and uterine cervix and myeloid leukemia.

Diagnoses Of Cancer Decline

The analysis found that the overall incidence of cancer began inching down in 1999, but not until the data for 2005 were analyzed was it clear that a long-term decline was underway. “The take-home message is that many of the things we’ve been telling people to do to be healthy have finally reached the point where we can say that they are working,” Brawley said. “These things are really starting to pay off.”

Brawley and others cautioned, however, that part of the reduction could be the result of fewer people getting screened for prostate and breast cancers. In addition, the rates at which many other types of cancer are being diagnosed are still increasing
…
Some experts said the drop was not surprising, noting that it was primarily the result of a fall in lung cancer because of declines in smoking that occurred decades ago. They criticized the ongoing focus on detecting and treating cancer and called for more focus on prevention.

“The whole cancer establishment has been focused on treatment, which has not been terribly productive,” said John C. Bailar III, who studies cancer trends at the National Academy of Sciences. “I think what people should conclude from this is we ought to be putting most of our resources where we know there has been progress, almost in spite of what we’ve done, and stop this single-minded focus on treatment.”

Google Flu Leading Indicator

Google Flu Trends

During the 2007-2008 flu season, an early version of Google Flu Trends was used to share results each week with the Epidemiology and Prevention Branch of the Influenza Division at CDC. Across each of the nine surveillance regions of the United States, we were able to accurately estimate current flu levels one to two weeks faster than published CDC reports.
…
So why bother with estimates from aggregated search queries? It turns out that traditional flu surveillance systems take 1-2 weeks to collect and release surveillance data, but Google search queries can be automatically counted very quickly. By making our flu estimates available each day, Google Flu Trends may provide an early-warning system for outbreaks of influenza.

For epidemiologists, this is an exciting development, because early detection of a disease outbreak can reduce the number of people affected. If a new strain of influenza virus emerges under certain conditions, a pandemic could emerge and cause millions of deaths (as happened, for example, in 1918). Our up-to-date influenza estimates may enable public health officials and health professionals to better respond to seasonal epidemics and — though we hope never to find out — pandemics.

This is an interesting example of finding new ways to quickly access what is happening in the world. Google must be doing significant amounts of similar things to see how usage patterns can server as a leading indicator.

Atlantic Hurricane Season 2008

photo of hurricane evacuation sign

Here is a nice post on weather and understanding data – Atlantic Hurricane Season 2008

A well-accepted metric which convolves storm frequency, intensity, and duration is called accumulate cyclone energy (ACE) and is calculated very simply: take the maximum sustained winds reported by the NHC every 6-hours for all storms (> 34 knots), square this value, and sum over the entire lifetime, then divide by 10,000. In 2007, even though there were also 15 storms, the ACE was only 72 compared to 132 for 2008 with the same number of named storms. This is partially because the storms in 2008 were much longer lived especially Bertha.
…
When encapsulated in the recent active period in North Atlantic activity (1995-2007), 2008 experienced normal or expected activity as measured by ACE. In terms of a long-term climatology, either the last 30 or 65 years, 2008 is clearly an above average year.

Data can’t lie but mistaken assumptions can lead you to form mistaken impressions. If you believe the number of named storms = hurricane activity and then are surprised that in fact there was many more days of hurricane activity it is not because the data lied but because you didn’t understand what the data represented.

William G. Hunter Award 2008: Ronald Does

The recipient of the 2008 William G. Hunter Award is Ronald Does. The Statistics Division of the American Society for Quality (ASQ) uses the attributes that characterize Bill Hunter’s (my father – John Hunter) career – consultant, educator for practitioners, communicator, and integrator of statistical thinking into other disciplines to decide the recipient. In his acceptance speech Ronald Does said:

The first advice I received from my new colleagues was to read the book by Box, Hunter and Hunter. The reason was clear. Because I was not familiar with industrial statistics I had to learn this from the authors who were really practicing statisticians. It took them years to write this landmark book.
…
For the past 15 years I have been the managing director of the Institute for Business and Industrial Statistics. This is a consultancy firm owned by the University of Amsterdam. The interaction between scientific research and the application of quality technology via our consultancy work is the core operating principle of the institute. This is reflected in the type of people that work for the institute, all of whom are young professionals having strong ambitions in both the academic world and in business and industry.
…
The kickoff conference attracted approximately 80 statisticians and statistical practitioners from all over Europe. ENBIS was officially founded in June 2001 as “an autonomous Society having as its objective the development and improvement of statistical methods, and their application, throughout Europe, all this in the widest sense of the words” Since the first meeting membership has grown to about 1300 from nearly all European countries.