The most important quality of a successful surgeon is judgement. In fact, many outcomes are determined by a surgeon before he or she begins operating — choosing an appropriate surgical candidate is often the most difficult decision a surgeon will make.

A good surgeon certainly has inherent physical talent and dexterity. Most great surgeons have exceptional skill with a scalpel.

In addition, patient volume is a huge determinant in surgical outcomes. A surgeon with a large-volume practice is much more adept at performing routine procedures and is also much more skilled at handling surprises during an operation.

Surgeons who have been well-trained and well-educated at academic institutions during their residencies and fellowships are more likely to practice evidence-based medicine and adhere to “best practices” and clinical guidelines.

Recently, an independent research group known as Pro Publica has received a great deal of attention for a newly published project that rates surgeons.

The project has been met with a great deal of scrutiny by the medical community, and has stirred significant debate since it was released in July 2015.

While it is a worthwhile goal to providing patients with an objective means of evaluating physicians, Pro Publica’s attempt has fallen significantly short — and if anything, has created more confusion and frustration among both doctors and patients.

Helping the public make informed decisions about healthcare is an essential part of improving the quality of care in the U.S. But the information provided to the public must be easily digestible, as well as reliable and accurate and supported by solid, well-conducted research.

The methodology by which data is assembled and meaningful conclusions are reached is critical in determining whether or not the conclusions are of any use to the intended audience.


What Is Pro Publica?
According to its website, Pro Publica is an “an independent, nonprofit newsroom that produces investigative journalism in the public interest.” It is funded by philanthropic donations as well as advertising dollars.

The organization is staffed by nearly 45 investigative journalists, and is located in New York City. Their mission is:

“To expose abuses of power and betrayals of the public trust by government, business, and other institutions, using the moral force of investigative journalism to spur reform through the sustained spotlighting of wrongdoing.”

With the surgeon rating project, Pro Publica sought to produce a mechanism by which healthcare consumers could examine outcomes data prior to choosing a medical professional.

From Medicare billing databases, the project collected information on 3,600 hospitals involving almost 17,000 surgeons. The outcomes of almost 64,000 Medicare patients were evaluated. A searchable “surgeon scorecard” was created online and has been available to the public since early July.

Data were limited to a few selected low risk procedures, and only included patients admitted to hospitals for these procedures. The analysis did not consider outpatient surgical center locations, nor did they evaluate data from patients admitted from emergency departments.

Doctors in particular specialties such as orthopedics, neurosurgery, general surgery, and urology were rated based on two outcomes — death and readmission — and all data were mined from a Medicare billing database.

The journalists clearly state their methodology on their website. They examined records from inpatient hospital stays from 2009-2013, and looked at complications from what they defined as low-risk surgeries.

While Pro Publica insists that they consulted with “experts” in each area evaluated, they do not mention if they had any guidance from statisticians or other experts in the design of medical research investigations.


Project Fundamentally Flawed
While well-intended, the project is of no practical use because its methodology was flawed at the beginning. Mining Medicare databases assumes that the data entered were accurate. But coding and data-entry errors occur every single day in hospitals, and the project does not take into consideration surgeons who accept patients that others would turn away due to high risk or other considerations.

In addition, the data analysis only takes into consideration inpatient procedures. Many of these low-risk procedures are performed in outpatient surgical centers, and including those numbers would certainly bolster a surgeon’s “scorecard.”

What’s more, the research presented by Pro Publica was not peer reviewed. Every reputable medical journal requires that investigations be reviewed by a minimum of two or three independent, anonymous experts in the discipline of interest. These experts are charged with evaluating not only the quality of the research and its findings, but also the methods that were utilized in the study.

Other than mentioning a group of physician experts that were consulted to identify complications, there is no mention of a peer-review process and no mention of methodology review.

Ultimately, this database unfairly evaluates surgeons. While not all physicians are created equal, the methodology involved in Pro Publica’s ratings has produced unreliable and inaccurate data. I fear that rating projects of this sort — that are not based on sound science — will result in many physicians refusing to treat sicker, higher-risk patients.

Along with many of my colleagues, I applaud any effort that can potentially empower patients to make informed decisions about their healthcare. However, I cannot endorse faulty science. If you put garbage data into a statistical program, you simply get well-analyzed garbage out.

As British economist Ronald Coase once said….”If you torture the data long enough, it will confess to anything…”