Don’t believe the hype: Today’s AI unlikely to best actual doctors at diagnosing patients from medical scans

Majority of academic studies into hospital image processing aren’t subjected to clinical testing

Don’t fall for the overblown claims that AI algorithms are just as good as, or even better, than human doctors at diagnosing diseases from medical images. That’s according to a study published in The British Medical Journal on Wednesday.

A group of researchers led by Imperial College London in the UK, studied 91 peer reviewed papers that applied deep learning algorithms, mostly convolutional neural networks, to look for common signs and symptoms of various illnesses from cancer to glaucoma. Ten studies were based on physical trials, whilst the rest of the 81 were purely academic.

The majority of these 81 papers, 69 in fact, all boasted about AI having superior or at least comparable performance compared to clinicians when applied to a particular problem, whether that’s spotting cancerous tumours in breast cancer or scarring in liver tissues for cirrhosis in liver damage. Only two admitted doctors were better than machines, and 14 said the machine learning models could aid them in diagnoses.

It’s no wonder that these types of studies are accompanied with splashy headlines claiming that computers are more accurate than real human doctors. But read the small print. Many of these papers might report impressive numbers, but the testing is often limited to the datasets that the researchers have compiled themselves. Only six of the 81 AI algorithms were applied on real patient data in clinical settings.

cough

Thankfully, our AI savior is here to nail the COVID-19 pandemic: A neural network that can detect coughing

READ MORE

The sample sizes used to train and test the models are often small. Hell, in some cases fake data is generated because it’s often difficult to obtain real data from patients due to privacy concerns. The average number of human experts pitted against the algorithms was four.

What’s more troubling is that these studies are often very difficult to replicate. Full access to the datasets fed to the machine learning models was unavailable in 95 per cent of the studies. The whole code describing the algorithms themselves was absent 93 per cent of the time.

The researchers reckon that around two thirds of the 81 studies were likely to be highly biased. Many of them were “non-randomised”, neglecting to take into account the effects of age, sex, and medical history of patients.

“We found only one randomised trial registered in the US despite at least 16 deep learning algorithms for medical imaging approved for marketing by the Food and Drug Administration (FDA),” according to the BMJ research.

Although deep learning is fancy and exciting, drawing in investors and developers in industry and academia, it’s still too premature to claim that it is better at medical screening than real healthcare professionals. Clinical trials often take years to carry out before drugs or medical devices are deemed effective.

“At present, many arguably exaggerated claims exist about equivalence with or superiority over clinicians, which presents a risk for patient safety and population health at the societal level, with AI algorithms applied in some cases to millions of patients,” the paper concluded.

“Overpromising language could mean that some studies might inadvertently mislead the media and the public, and potentially lead to the provision of inappropriate care that does not align with patients’ best interests.”

The researchers aren’t completely down in the dumps over medical AI technology, however. Mahiben Maruthappu, co-author of the study and CEO of Cera Care, a startup focused on healthcare for the elderly, told The Register.

“Machine learning, when developed in a robust manner and well evaluated, can be transformative for many parts of healthcare, from how we triage patients in A&E, to diagnosis, to recommendations on prescriptions, to advice to patients on lifestyle modification,” he said.

“At a time when health systems face unprecedented pressure, such solutions could be invaluable, when delivered in a safe and effective manner.” ®

Sponsored:
Practical tips for Office 365 tenant-to-tenant migration

Articles You May Like

Quibi Review: Bite-Sized Content, Terrible Pricing
Asus ROG Zephyrus Duo 15 Dual-Screen Gaming Laptop With 10th Gen Core i9, Nvidia GeForce RTX Super Launched
Researchers to study if startup’s wrist-worn wearable can detect early COVID-19 respiratory issues
IRL pivots into virtual event calendar In Remote Life
Nvidia GeForce RTX 2080 Super, 2070 Super for Gaming Laptops Launched, Max-Q Update Announced

Leave a Reply

Your email address will not be published. Required fields are marked *