External validation of the ACC/AHA ASCVD risk score in a Colombian population cohort

Purpose of the study

This study aims to evaluate the behavior of the ACC/AHA ASCVD risk score in terms of discrimination and calibration to predict cardiovascular risk in a primary prevention cohort sample followed in a Colombian hospital.

study design

This retrospective cohort study was conducted from July 2021 to May 2022 in patients without established cardiovascular disease evaluated at the Hospital Universitario Fundación Santa Fe de Bogotá (FSFB) between 2009 and 2010. FSFB is a tertiary care center located in Bogotá , the capital of Colombia that provides care to patients from different parts of the country, mainly private health insurance.

The risk of atherosclerotic cardiovascular events was calculated with baseline data and subsequently the occurrence of major cardiovascular events within ten years was observed. The study was approved by the ethics committee of the Fundación Santa Fe de Bogotá. Our study was carried out in accordance with the Declaration of Helsinki and established regulations. Informed consent was obtained from all participants before including them in the study.

The data source

Data from two cohorts of patients seen in outpatient special care programs were used. The first cohort (intermediate/high risk) corresponds to a special care program composed of 421 patients belonging to the hospital’s hypertension and diabetes program in 2009 and 2010. The second cohort (low risk) is composed of 2334 patients seen in the FSFB executive selection program in 2009 and 2010.

According to the study’s inclusion criteria, patients had to be enrolled in the Hypertension and Diabetes Program or the Executive Screening Program (a preventive medical screening service available to the general population) in 2009 or 2010. The clinical history of each of the patients in these two programs was reviewed to determine the first visit to the FSFB, where sufficient data was collected to estimate their cardiovascular risk using the ACC/AHA ASCVD score. The estimation of cardiovascular risk was only carried out in patients meeting the eligibility criteria. Subsequently, the medical files of the patients included were examined in order to verify whether they had suffered major cardiovascular events in the ten years following the date of the risk calculation.

Patients who did not have 10-year follow-up data were contacted by phone or email to inquire about the occurrence of cardiovascular events during the specified 10-year period. All information collected from both cohorts was stored in REDCap. Access to this record, as well as medical records, was restricted to study investigators.

Speakers

Patients were selected in the two cohorts based on the following inclusion criteria: Colombian patients aged 40 to 79 years, with no history of major cardiovascular events, with LDL-c levels below 190 mg/dl and who had sufficient information to calculate their cardiovascular risk at 10 years. Exclusion criteria were: patients whose cardiovascular events during the 10 years following the risk calculation could not be verified, evidence of other forms of prior atherosclerotic cardiovascular disease (transient ischemic attack, unstable angina, angioplasty, coronary artery bypass graft (CABG), or peripheral arterial disease) at the time of the risk calculation, or use of statins at the time of the risk calculation. Patients taking statins were excluded to avoid bias in the calculation of cardiovascular risk by lowering non-HDL cholesterol levels, thus generating a change in the estimated initial risk. Consecutive non-probability sampling was performed, including all subjects from databases meeting the inclusion criteria. Only patients with complete information for the risk calculation and a follow-up of 10 years were included for the analysis.

Results

The primary endpoint was the occurrence of a major cardiovascular event. Major cardiovascular events included were fatal and non-fatal acute myocardial infarction (AMI) according to the fourth universal definition of AMI²², and fatal and non-fatal stroke (signs and symptoms of neurologic deficit, compatible CT and/or cerebral nuclear magnetic resonance findings, or need for endovascular interventions). Outcome information was collected from medical records and analyzed separately by each of the investigators who assessed the validity of the diagnosis according to the previous definitions. Any discrepancies during the process were discussed by all authors and only patients eligible for the study by consensus of all authors were included.

Independent variables

The independent variables were: age (at the date of the risk calculation), sex, race (Afro-Colombian vs white/other), total cholesterol, HDL cholesterol, systolic blood pressure, treatment for high blood pressure (Yes/No), smoking (Yes/No) and diabetes mellitus (Yes/No).

Sample size

According to the recommendation made by authors like Carretero, it takes between 5 and 10 people for each item that makes up the score²³. In this case, the ACC/AHA ASCVD score is composed of nine items. Therefore, a total of 90 participants is required. This number was greatly exceeded for both cohorts. Patients who did not have complete information were excluded, so there are no missing variables of interest.

statistical analyzes

First, a descriptive analysis of the included population was carried out using the Stata17® program.²⁴. Continuous variables are reported as means and standard deviations, while categorical variables are reported as frequencies and proportions. The characteristics of the total population, as well as of each cohort separately, were described. The official risk calculator available on the American Heart Association website was used to calculate cardiovascular risk https://tools.acc.org/ldl/ascvd_risk_estimator/index.html#!/calulate/estimator/.

Calibration (coincidence between the expected and observed probability of cardiovascular events in the population) was assessed using a multivariate logistic regression model in which the predictor variables were those included in the original model (age, sex , race, total cholesterol values, HDL cholesterol levels, systolic blood pressure, use of antihypertensive medication, smoking, and diagnosis of diabetes mellitus). The interaction between these variables was assessed at the time of model analysis. A penalized regression was used so as not to exclude data that had fewer observations in the model (eg, Afro-Colombian race). Subsequently, predicted events were compared to observed events in four different risk categories (< 5 %, 5 à 7,5 %, 7,5 à 20 % et > 20%) based on AHA guideline thresholds.²¹. A Hosmer Lemeshow (HL) test was used to assess the goodness of fit of the model. This chi-square based test is used for binary response variables (cardiovascular event or not) and assesses how well the data fits the model. A small HL number, usually associated with a p-value greater than 0.05, means that there is no statistically significant difference between the expected and observed events. In this case, the overall fit of the model is good. By choosing only 4 subgroups, the power of the HL test is greater than it could be if the subgroups were risk deciles. Also, in samples above 400, this test has good power to find a bad calibration.

Discrimination (the ability of the score to differentiate individuals who will have a cardiovascular event from those who will not) was assessed using the C statistic, or area under the ROC curve. An AC stat above 0.75 is considered good. The results of sensitivity, specificity and predictive values ​​were analyzed for a threshold of 20%, which corresponds to the classification limit for high-risk patients.

The results obtained were analyzed for the total population of the two cohorts. Score behavior was also analyzed in each cohort separately (intermediate/high risk vs low risk). Finally, a sensitivity analysis was performed with the race variable to assess possible changes in the diagnostic performance of the score.