When COVID-19 first reached our shores, Public Healthcare Institutions (PHIs) searched for ways to fortify their preparations and responses. Changi General Hospital (CGH) reached out to healthtech experts from IHiS to explore the possibility of developing an artificial intelligence (AI) predictive model known as Community Acquired Pneumonia and COVID-19 AI Predictive Engine (CAPE) that can generate a risk score for pneumonia patients.

As pneumonia is one of the key symptoms of deteriorating cases of COVID-19, the model would potentially enable more timely triaging and treatment, and defer or mitigate a patient’s likelihood of being admitted into the intensive care unit. Such a predictive model would serve as a decision support for doctors, or in certain cases, even bring new insights where doctors might have missed, especially for patients with risk factors.

Dr Charlene Liew, Deputy Chief Medical Informatics Officer, CGH, and Director of Innovation, SingHealth Radiological Sciences Academic Clinical Programme (RADSC ACP), who initiated the collaboration with IHiS, said that this computer assisted tool would help to prioritise treatment and escalate the monitoring of patients who are likely to become unstable or critically ill. This has been identified as a research priority by the WHO (World Health Organization). She added:

“Previously, based on Chest X-Ray (CXR) imaging and analysis using human vision, we were limited to being able to say if pneumonia is present. There was also no method of identifying which COVID-19 patients were at risk of requiring critical care. So one of the main advantages of using CXR as a predictive and prognostic tool when combined with artificial intelligence (AI) is that the risk can be calculated almost instantaneously, without relying on multiple blood tests and an extensive assessment of past medical history.

With CAPE’S AI predictive model, we can augment human intelligence and vision, so that the Emergency Department and ward doctors can receive an early warning for clinical deterioration and prescribe the appropriate interim measures to improve patient outcomes.”

Bringing HealthTech AI to Healthcare

IHiS and CGH are long time partners in bridging healthcare and healthtech innovation. The strong ties in the creation of innovation proved to be valuable in working together at speed in the face of a crisis. CGH and IHiS saw the potential in using Artificial Intelligence (AI) Technology in the combat of COVID-19 and quickly formed a multidisciplinary team committed to the success of the project.

Leveraging past experience in implementing large scale machine learning application, the IHiS Health Insights team conceptualised the end-to-end solution covering data pre-processing, model development and redesign of workflow.

Agile and design thinking methodologies were used to accelerate project planning and implementation as the team iteratively improved the Artificial Intelligence (AI) models with clinical inputs. The implementation approach was to have a quick adaptation of CAPE application to the existing radiology workflow with shorter deployment turnaround.

Andy Ta, Director (Health Insights), Emerging Capabilities, IHiS, said:

“A model of such complexity would normally take longer to develop. However, due to the urgency and potential impact for COVID-19, both CGH and IHiS mobilized a strong and committed team and adopted the rapid development approach on-site to accelerate the project.”

Developing the CAPE Predictive Engine

Pair coding approach was used for model development where six healthtech data scientists from IHiS broke into two teams to do concurrent coding. Using different deep learning frameworks, the two teams set out to develop the same Artificial Intelligence (AI) components for CAPE which ensured the robustness of the model with consistent outcomes achieved. Best practices from the different frameworks were adopted to form the state of the art solution.

Dr Goh Han Leong, Principal Specialist, Emerging Capabilities-Health Insight, IHiS said:

“All of us came together with a clear goal – to work as one team to complete this project within the shortest timeline possible, from data mining to coding and approvals. The tight timeline made the task seemingly impossible but we managed to pull it off by tweaking our existing model development framework with pair coding and adopting KISS – Keep It Simple, Stupid – principles. We are very thankful for the trust and support that CGH clinicians have given to make this new way of working possible.”

Using the Past to Predict the Future with CAPE

Most of the current assessment tools in the market use either radiology images or Electronic Medical Records (EMR) data. However, CAPE is equipped with the capabilities to utilise both through a single end-to-end machine learning model. Machine learning is essentially a set of algorithms which use statistics to find patterns in massive amounts of data.

To get the machine to learn how to predict an accurate score, the team fed it with historical data – more than 3,000 CXR images and 200,000 data points – to analyse what happened in the past to predict what is likely to happen in future similar cases. After two months of development, which included ensuring the viability of the predictive engine through many rounds of training, validating and testing the various permutations of data, three models were developed.

Model 1: Patient likely to be discharged in two days or less

Model 2: Patient at risk of inpatient mortality

Model 3: Patient at risk of being admitted to the ICU

The models help clinicians to determine the likelihood of whether a patient has mild or severe pneumonia. This knowledge allows them to prioritise clinical worklists; when the AI predicts a high-risk prognosis, more medical attention can be channelled to these patients. It also provides an early alert to any deteriorating prognosis.

After extensive prospective validation on site, all three models were found to have good predictive power of an “Area Under Curve” (AUC) of between 0.71 to 0.83. Statistically speaking, the closer the AUC is to 1.0, the more accurate its prediction is.

To ensure ease of integration and use, CGH and IHiS incorporated CAPE into the radiology workflow as an application with only minor changes to the existing system. Ongoing work includes integrating more historical data-sets into the predictive engines, which would likely make it even more accurate.

Case Study of CAPE Application

A 42-year old man tested positive for COVID-19. CAPE calculated that the likelihood of his condition requiring a stay of more than 2 days as 80.6%. It was proven accurate when the man was subsequently admitted to ICU for deteriorating oxygen saturation.

The CGH team is also working to validate the model in other public health institutions in Singapore to improve the robustness of the model. The team is also exploring collaborative models, including uploading it as a “freeware” collaborative tool on a research platform for interested researchers.

View related articles: CAPE Artificial Intelligence Tool Predicts Pneumonia Severity in Covid Patients