Machine Learning in Prognostic Decision-Support Systems Study - Computer Science Perspective

Data

The model was trained using the Meta-analysis Global Group in Chronic Heart Failure (MAGGIC) data-set, which compiled patient data from 30 studies including both randomized clinical trials and observational registries.

The data set included 30,389 patients who have experienced a heart failure event. Of these, 18.8% died within 1 year of experiencing heart failure.

Note that all risk scores reported are for patients who have already experienced heart failure.

Of the 30,389 patients, 66% were male. The average age was 68, with a standard deviation of 12 years.

31 patient characteristics, symptoms, and other features were collected, including:

• Basic patient details: Age, gender, ethnicity
• Patient habits: BMI, Smoking, Diabetes condition
• Physical symptoms: Shortness of breath (at rest and exercise), rales
• Relevant prescriptions: ACE inhibitors/ARB, Beta blockers
• Data and test results: Systolic and diastolic blood pressure, ejection fraction, etc.
• New York Heart Association Score (Class I-IV) - click here for details

Missing values were imputed using standard methods (Multiple Imputation by Chained Equations), which provide accurate estimates for missing or not-recorded patient symptoms and characteristics

More details about the MAGGIC data-set, including a full list of features, can be found in (Pocock et al., 2012).

Model Training and Implementation

The model utilised was a Neural Network, a sophisticated deep learning algorithm that has seen increased attention in recent years, as multiple studies have shown it to vastly outperform simpler linear models and other methods that have been traditionally utilised to generate risk scores in medicine.

A neural network is a Machine Learning algorithm inspired by the human brain, utilizing connected nodes organised in hidden layers. With a single node and layer, the network would be essentially equivalent to a the linear predictive model known as Logistic Regression. However, adding nodes and layers enables the network to learn more complex, non-linear patterns in the data.

The network uses optimization techniques and non-linear activation functions to automatically discover these complex patterns and learn how to make predictions from data. You can find more information here.

Below is a generic image of a neural network by Machine Learning expert Michael Nielsen:

The network takes in a large amount of data, in the form of patient characteristics or features, and outputs the probability of a patient dying within 1 year of a heart failure event.

The Neural Network we utilised contained 2 hidden layers of 100 and 20 nodes respectively. In general, as explained here, network architecture, including the number of nodes and layers, must be tweaked and tuned to ensure the network is large enough to capture complex data patterns, but not too large compared to the size of the data-set, which would result in over-fitting and poor predictive performance in practice. The architecture we have selected of 2 layers with 100 and 20 nodes, respectively, was tuned and found to achieve highly accurate results.

The model was trained using 10-fold cross validation, using 90% of the data for training and the rest for testing. This was repeated 10 times, and we report average figures over the run. For more information on cross-validation, click here.

Model Accuracy

The neural network showcases 81.26% accuracy, which means that 81% of the test cases were classified correctly by the model.

The model's AUC-ROC (area under the receiver operating curve), which is equal to the C-Index, is 0.725, with a standard deviation of 0.0054, averaged over 10 runs. This suggests high accuracy for the type of data, as we demonstrate below by comparing our network to a simple linear model and a risk score currently in use. For more information on the C-Statistic or AUC-ROC, click here.

The model's ROC curve is below:

In addition, the model achieves an AUC-PR (area under the precision recall curve) of 0.376. For more information about precision and recall, click here.

The neural network's accuracy values for both AUC-ROC and AUC-PR are much higher than the best linear regression model trained on the same data. They are also significantly higher than the accuracy of the MAGGIC Risk Score, an accepted risk score trained on the same data and developed by the European Society of Cardiology and other researchers. For information on the MAGGIC Risk Score, click here. The chart below summarises these results:

Model Interpretation: Stratified Linear Approximation

We have trained a neural network to predict the risk, or probability that a given patient who has suffered heart failure will die within one year of the heart failure event.

The model was trained on the data-set discussed previously, containing over 30K patients and 31 patient characteristics and features ranging from age to their New York Heart Association Score.

As shown previously, the neural-network outperforms simple linear models. However, the network is a black-box, which means it is impossible to tell how patient characteristics contribute to the outputted risk scores it produces.

In order to better understand how the model produces a prediction, black-box models, like neural networks, can be approximated locally using simpler models, including linear regression models.

To create local approximations, after the neural network generated its risk scores, a different linear model was trained for each quintile of risk: 0-20%, 20-40%, etc.

The Root Mean-Squared Error between the neural network model and the linear approximation is 0.036, suggesting the 2 models are very close.

Below is a summary of the coefficients for each local linear model:

Since the linear approximation and the neural network models are so close, the value of each characteristic’s coefficient closely approximates their contribution to the variability in predicted patient risk score.

For instance, the negative coefficient for Beta blocker prescriptions suggests a negative correlation between this variable and risk of death, though the magnitude differs among risk strata. Some factors, such as shortness of breath at rest or gender, are only significant for high risk groups.

Model Interpretation: Decision Tree

We have trained a neural network to predict the risk, or probability that a given patient who has suffered heart failure will die within one year of the heart failure event.

The model was trained on the data-set discussed previously, containing over 30K patients and 31 patient characteristics and features ranging from age to their New York Heart Association Score.

As shown previously, the neural-network outperforms simple linear models. However, the network is a black-box, which means it is impossible to tell how patient characteristics contribute to the outputted risk scores it produces.

In order to better understand how the model produces a prediction, black-box models, like neural networks, can be approximated using simpler models, such as decision trees.

To create local approximations, after the neural network generated its risk scores, a different decision tree was trained for each quintile of risk: 0-20%, 20-40%, etc.

The Root Mean-Squared Error between the neural network model and the decision tree approximation is 0.046, suggesting the 2 models are very close.

Click below for a summary of the depth-3 trees for each local linear model:

Note that each risk stratum utilises different patient characteristics for classification. Few patients are classified in the 80-100% class.

Since the models are so close, we can view the trees as an approximation explaining how each patient is assigned a risk score based on their characteristics and symptoms.