With the development of a well-organized colorectal cancer (CRC) screening program and an increasing population adhering to it, the number of patients diagnosed with early-stage CRC has risen.¹ Early CRC is defined as the invasion of cancer cells into the mucosal or submucosal layer, regardless of lymph node metastasis (LNM). It can be treated using endoscopic methods instead of radical surgery if the risk of LNM is negligible.^2-4 Nevertheless, when early CRC is treated by endoscopic resection, there is a pitfall: the presence or absence of LNM cannot be histologically verified. Hence, if there are one or more high-risk pathological features suggestive of LNM, additional surgery with lymph node dissection is recommended after endoscopic resection.^, The guidelines suggested by the Japanese Society for Cancer of the Colon and Rectum (JSCCR) state the following pathological findings as the high-risk features for LNM: (1) depth of submucosal invasion ≥1,000 μm; (2) lymphovascular invasion positive; (3) poorly differentiated adenocarcinoma, signet-ring cell carcinoma, or mucinous carcinoma; and (4) budding grade of 2 or 3.

However, when an additional surgery was performed based on these recommendations and the surgically resected specimens were analyzed, LNM was confirmed only in 8.8% to 14.3% of cases.^6-9 In other words, about nine out of 10 patients underwent unnecessary surgery. Furthermore, surgery-related mortality and morbidity are still not negligible despite advances in surgical techniques. Therefore, there is a need to develop a high-accuracy preoperative predictive model for LNM in early CRC.¹⁰ To meet this requirement, several attempts have been made to develop an artificial intelligence (AI)-based decision-making model for predicting LNM in early CRC.^11,12 These models demonstrated superior accuracy in predicting LNM in early CRC compared to the current guidelines. However, most of these models were created based on the patient’s clinical and pathological data, excluding endoscopic findings.

In this study, we aimed to develop an AI-based LNM predictive model that focuses on both the histological features and the endoscopic characteristics of early CRC. We also evaluated whether the new AI model demonstrated good accuracy in LNM prediction compared to the JSCCR guidelines.

1. Study design and population

This was a retrospective single-center study with the development and cross-validation of an AI model. In this study, patients who underwent surgical resection of T1 CRC, with or without prior endoscopic resection at Asan Medical Center, Seoul, Korea, from 2011 to 2018, were included. Furthermore, patients who had undergone endoscopic resection at other institutions followed by surgery at Asan Medical Center were also included. Among these, the following patients were excluded: (1) those with synchronous CRC; (2) those with non-radical surgery; and (3) those with missing values, such as poor-quality endoscopic images. We gathered demographic variables and clinical information, including age, sex, body mass index, family history of CRC, medication, smoking and alcohol history, laboratory data (fasting blood glucose, total cholesterol, and serum carcinoembryonic antigen levels), pathological findings of the surgical specimen, and endoscopic images.

2. Pathological findings

The pathological findings of surgical specimens were evaluated by board-certified pathologists in our institution. The following pathological factors were included in developing an AI model to predict LNM for T1 CRC: (1) tumor histology, (2) lymphovascular invasion, (3) tumor budding, and (4) invasion depth of submucosa (SM). We defined deep SM invasion as the absolute depth of SM invasion >1,000 μm, Haggitt level 4, or Kikuchi SM level 2−3.^13,14

3. Endoscopic findings

The endoscopic images were evaluated independently by two board-certified, experienced endoscopists (S.W.H. and J.S.B.). All the analyzed images were conventional white-light endoscopy images. The typical endoscopic findings suggesting deep SM invasion included in the analysis were as follows: (1) ulceration/depression, (2) expansion, (3) white spots, and (4) hardness.¹⁵ In instances of disagreement between the two endoscopists, a consensus was reached through team discussion. Other endoscopic findings, such as tumor size, location (colon or rectum), morphology (pedunculated or non-pedunculated), and pit pattern by Kudo classification, were also analyzed.¹⁶

4. Outcome definitions

The primary outcome of this study was the clinical validation following the development of an AI model for predicting LNM in T1 CRC. The stratified 5-fold cross-validation was implemented to validate the model’s performance under generalized circumstances. Receiver operating characteristics curves were plotted and the area under the receiver operating characteristics curve (AUROC) was measured to compare our model’s discriminating power with that of the Japanese guidelines.

5. Subanalysis

We compared the diagnostic performance according to the tumor location (colon vs rectum) because rectal cancer was reported to have a higher risk of LNM as compared to colon cancer.¹⁷ Furthermore, as the initial endoscopic resection group serves as an actual prediction target for our model in clinical practice, we assessed the diagnostic performance within patients who underwent endoscopic resection followed by surgical resection.

6. AI model development

The predictive models were developed using four supervised AI algorithms, which included a regularized logistic regression classifier (RLRC), random forest classifier (RFC), CatBoost classifier (CBC), and the voting classifier (VC) created with the three other models by a 2:1:4 ratio of votes. RLRC, which adds a penalty to the loss function to reduce the possibility of overfitting to the development dataset, is anticipated to demonstrate optimal performance for unseen data.¹⁸ RFC method allows the diversity of decision trees to improve classification performance while preventing overfitting to the development set, creating a model that also works well with unseen data.¹⁹ CBC exhibits outstanding performance among gradient-boosting algorithms, especially in tasks involving numerous categorical variables.²⁰ VC tends to generate robust predictions, even in cases where one or two classifiers make errors. The classification probability of VC with well-calibrated classifiers also approaches the true probability.²¹

We utilized the previously mentioned four pathological and four endoscopic findings for developing the prediction model: tumor histology, lymphovascular invasion, tumor budding, depth of SM invasion, hardness, expansion, white spot, and ulceration/depression. The best-performing model among the four algorithms was chosen as the final model for subanalysis.

7. Statistical analysis

Continuous variables were described as mean±standard deviation or median (interquartile range), while categorical variables were expressed as frequency and percentage. The Mann-Whitney U tests were employed to compare continuous variables, and chi-square or Fisher exact tests were utilized for comparing categorical variables. AUROC was compared using DeLong’s test.²² All statistical tests were conducted using a two-sided approach, and a p-value <0.05 was considered statistically significant. The missing values were filled via Multivariate Imputation by Chained Equations imputation.²³ All analyses were performed using R software (version 4.0.2; R Foundation for Statistical Computing, Vienna, Austria).

8. Ethical statements

This study was approved by the Institutional Review Board at Asan Medical Center (IRB number: 2020−1676). The study was conducted by the Helsinki Declaration. Because of the retrospective nature of the study, the patient consent requirement was waived.

1. Baseline characteristics of the patients

We initially identified 1,635 patients with T1 CRC who underwent surgical resection at Asan Medical Center. After excluding patients based on the exclusion criteria, 1,386 patients were ultimately selected as the study population, with 173 of them having LNM (12.5%) (Fig. 1). Baseline characteristics are demonstrated according to LNM in Table 1. In the LNM group, 64 patients (37.0%) underwent endoscopic plus surgical resection, and 109 patients (63.0%) underwent initial surgical resection. Endoscopic and pathological features of T1 CRC patients with and without LNM were summarized in Table 2. We used four pathological variables (tumor histology, lymphovascular invasion, tumor budding, and depth of invasion) and three endoscopic variables (hardness, white spot, and ulceration/depression), that exhibited statistically significant differences between the two groups to develop the prediction model for LNM. Although not statistically significant, another endoscopic factor, expansion, was included in the development of the prediction model based on its clinical significance in previous studies.¹⁵

Figure 1. Flowchart of the included patients. CRC, colorectal cancer; LNM, lymph node metastasis.

Table 1 . Baseline Characteristics.

Variable	Patients with LNM (n=173)	Patients without LNM (n=1,213)	p-value
Age, yr	59.9±10.3	60.7±10.5	0.375
Sex			0.301
Male	98 (56.6)	741 (61.1)
Female	75 (43.4)	472 (38.9)
BMI, kg/m2	24.3 (22.6–26.2)	24.3 (22.3–26.3)	0.651
Family history of CRC			0.754
No	159 (91.9)	1,102 (90.8)
Yes	14 (8.1)	111 (9.2)
Use of medications
Aspirin	11 (6.36)	158 (13.0)	0.017
Statin	13 (7.51)	150 (12.4)	0.084
Alcohol consumption			0.350
Current or ex-drinker	85 (49.1)	646 (53.3)
Nonuser	88 (50.9)	567 (46.7)
Smoking history			0.011
Never	113 (65.3)	665 (54.8)
Ex-smoker	53 (30.6)	434 (35.8)
Current smoker	7 (4.05)	114 (9.4)
Fasting blood glucose, mg/dL	127 (104–154)	130 (107–157)	0.495
Total cholesterol, mg/dL	162 (140–186)	165 (140–189)	0.469
Serum CEA level, mg/dL	1.4 (0.91–2.0)	1.5 (0.99–2.1)	0.348
Location of T1 cancer			0.458
Colon	121 (69.9)	810 (66.8)
Rectum	52 (30.1)	403 (33.2)
Treatment modality			0.005
Surgery only	109 (63.0)	623 (51.4)
Endoscopic resection followed by surgery	64 (37.0)	590 (48.6)

Data are presented as mean±SD, number (%), or median (range)..

LNM, lymph node metastasis; BMI, body mass index; CRC, colorectal cancer; CEA, carcinoembryonic antigen..

Table 2 . Endoscopic and Pathological Features.

Variable	Patients with LNM (n=173)	Patients without LNM (n=1,213)	p-value
Tumor size, mm	18 (13–24)	18 (12–25)	0.938
Morphology			0.809
Nonpedunculated	125 (72.3)	891 (73.5)
Pedunculated	48 (27.7)	322 (26.5)
Hardness	152 (87.9)	925 (76.3)	<0.001
Expansion	99 (57.2)	685 (56.5)	0.925
White spot	86 (49.7)	466 (38.4)	0.006
Ulceration/depression	101 (58.4)	543 (44.8)	0.001
Tumor histology			<0.001
WD	35 (20.2)	474 (39.1)
MD	122 (70.5)	702 (57.9)
PD	9 (5.2)	21 (1.73)
SRCC	3 (1.7)	1 (0.1)
Mucinous adenocarcinoma	3 (1.7)	12 (1.0)
Others	1 (0.6)	3 (0.2)
Tumor budding			0.031
Grade 1	126 (72.8)	976 (80.5)
Grade 2 or 3	44 (25.4)	222 (18.3)
Depth of SM invasion			0.01
Deep	136 (78.6)	833 (68.7)
Superficial	37 (21.4)	380 (31.3)
Lymphovascular invasion			<0.001
Negative	97 (56.1)	991 (81.7)
Positive	76 (43.9)	222 (18.3)

Data are presented as median (range) or number (%)..

LNM, lymph node metastasis; WD, well differentiated; MD, moderately differentiated; PD, poorly differentiated; SRCC, signet-ring cell carcinoma; SM, submucosa..

2. Clinical validation of the developed model

The confusion matrix obtained through the 5-fold cross-validation was presented in Supplementary Fig. 1. The AUROC of RLRC, RFC, CBC, and VC models for LNM prediction were 0.673 (95% confidence interval [CI], 0.574 to 0.772), 0.639 (95% CI, 0.542 to 0.736), 0.679 (95% CI, 0.582 to 0.775), and 0.677 (95% CI, 0.580 to 0.774), respectively. All models demonstrated superior performance compared to the AUROC of the JSCCR guidelines (0.525; 95% CI, 0.490 to 0.560) (p<0.001 between RLRC and JSCCR, p=0.001 between RFC and JSCCR, p<0.001 between CBC and JSCCR, and p<0.001 between VC and JSCCR) (Fig. 2). Other performance indicators such as accuracy, sensitivity, specificity, positive and negative predictive values were presented in the Supplementary Table 1. Lymphovascular invasion was the most influential factor for LNM in the SHAP beeswarm plot (Supplementary Fig. 2).

Figure 2. Receiver operating characteristics (ROC) curves for the artificial intelligence models and Japanese guidelines in predicting lymph node metastasis. AUROC, area under the ROC curve; CI, confidence interval; JpnGL, Japanese guideline; RLRC, regularized logistic regression classifier; RFC, random forest classifier; CBC, CatBoost classifier; VC, voting classifier.

3. Subanalysis according to the tumor location and treatment modality

We used the CBC model, which showed the best performance, as a tool for subgroup analysis. In the subgroups of T1 colon and rectal cancer, the AUROC for LNM prediction was 0.718 (95% CI, 0.600 to 0.836) and 0.615 (95% CI, 0.498 to 0.731) (Fig. 3). Moreover, there was no significant difference in the AUROC for LNM prediction between the patients who underwent initial endoscopic resection followed by surgery and those who underwent initial surgery (0.581 [95% CI, 0.473 to 0.688] vs 0.746 [95% CI, 0.636 to 0.855], p=0.845) (Fig. 4).

Figure 3. Receiver operating characteristics (ROC) curves for the CatBoost classifier model in the colon and rectum subgroups in predicting lymph node metastasis. AUROC, area under the ROC curve; CI, confidence interval.

Figure 4. Receiver operating characteristics (ROC) curves for the CatBoost classifier model in the initial endoscopic resection and initial surgery subgroups in predicting lymph node metastasis. AUROC, area under the ROC curve; CI, confidence interval.

Accurate prediction of the presence of LNM in patients who have undergone endoscopic resection of T1 CRC is a clinically important issue for deciding on additional surgery. Despite current guidelines recommending additional surgery if the endoscopic resection specimen exhibits unfavorable pathological features, 85.7% to 91.2% of the patients who underwent surgery, showed no LNM in the surgical resection specimen.^6-9,11,24 In this study, we discovered that AI models incorporating both endoscopic and pathological findings of T1 CRC were superior in predicting LNM compared to the JSCCR guidelines.

Several studies have attempted to accurately predict LNM in T1 CRC using AI algorithms. A Japanese study developed an AI model using the patient’s age, gender, tumor size, location, morphology, lymphovascular invasion, and histological grade.¹² This model demonstrated superior predictive accuracy compared to decisions based on the current U.S. guidelines.¹² In another study, a prediction model was proposed based on five histological findings, demonstrating a high predictive capability for LNM.²⁵ The JSCCR guidelines are the most commonly used and currently recommended for decision-making in CRC. Nevertheless, our study demonstrated that AI models can predict LNM in T1 CRC more accurately than the JSCCR guidelines. An important aspect deserving attention in our AI models is the development of algorithm that incorporate not only pathological features but also endoscopic findings. This differentiation provides a distinct advantage compared to previous studies. While we did not directly compare an AI model trained with endoscopic findings to one without, we suggest that AI models benefiting from diverse training data may exhibit superior performance compared to those with limited training data. However, such assumptions require further validation as there is a study that reported an AUROC of 0.83 based solely on histologic risk factors, which is higher than the AUROC of our models.²⁵ It should be noted that this study was a predictive model targeting the specific group of pedunculated T1 CRC, making direct comparison with our study, which included the entire T1 CRC population, difficult. Therefore, further research, encompassing not only pathological data but also demographic and clinical data such as medication history including aspirin, should be conducted to enhance the quality of AI models predicting LNM in T1 CRC. Additionally, a more comprehensive AI algorithm should be developed by including other endoscopic characteristics such as tumor size and location, in addition to the four endoscopic features (hardness, white spot, ulceration/depression, and expansion) used in our study.

In subgroup analysis, our AI model demonstrated high predictive power for LNM regardless of the location of T1 CRC. A previous study reported a significantly higher local recurrence rate in patients with T1 rectal cancer compared to those with T1 cancer in the colon proximal to the rectum when treated with only endoscopic resection.²⁶ Furthermore, bowel dysfunction after radical surgery for rectal cancer significantly impacts the quality of life.²⁷ Therefore, precise prediction of LNM in T1 rectal cancer is crucial. Our AI model revealed superior performance in the subgroup of T1 rectal cancer, similar to its performance in T1 colon cancer (Fig. 3). Thus, the AI model can be used for predicting LNM not only in T1 colon cancer but also in T1 rectal cancer, where precise LNM prediction is critical.

Reliable prediction of LNM in T1 CRC is necessary for determining the need for additional surgery after initial endoscopic resection in daily clinical practice. In our study, the AI model exhibited no significant difference in the performance of LNM prediction in T1 CRC between the initial endoscopic resection and initial surgery groups. Hence, we propose that the AI model may be readily applicable to patients who initially underwent endoscopic resection of T1 CRC to determine the necessity of additional surgery with greater confidence compared to the JSCCR guidelines. Nevertheless, although the difference was not statistically significant, the AUROC of 0.582 for the initial endoscopic resection group was numerically lower compared to 0.745 for the initial surgery group. Additionally, an AUROC of 0.582 indicates that the AI algorithm is insufficient for practical clinical application. Therefore, efforts are needed to improve performance by using larger training data and carefully considering reliable risk features, aiming to develop an AI algorithm that can be practically applied in determining the need for additional surgery in the initial endoscopic resection group.

We developed and evaluated four AI models. Among these, the CBC model showed the best performance. The CBC model has its advantage in tasks with many categorical variables.²⁰ The data of our study consisted mostly of categorical variables. Hence, in our study, the CBC model might outperform other AI models. Nevertheless, given the complexity of LNM occurrence in T1 CRC, additional efforts should be undertaken to explore the possibility of combining and employing ensembles of various AI models to improve the predictability of LNM in T1 CRC.

Our study has several limitations. First, it is a single-center, retrospective cohort study. Second, we did not train the AI algorithms directly with the colonoscopy images but employed the interpretation of two endoscopists. Hence, there might be interobserver or intraobserver bias. Finally, validation was performed only in a relatively small cohort without the temporal validation although separation of the patient cohort into the training and validation groups according to the sample collection period could have produced more robust results. However, to address this limitation, we performed the stratified 5-fold cross-validation, ensuring that the patients in the training and validation sets did not overlap. Additionally, to further verify the performances of our models, we performed 1,000 bootstrap iterations, another reliable validation method.²⁸ The results from bootstrapping were consistent with those from the 5-fold cross-validation (Supplementary Fig. 3), supporting the reliability of our AI models. Hence, we believe that our study holds significance, as it represents, to the best of our knowledge, the first clinical validation of AI models for predicting LNM incorporating not only pathological features but also endoscopic findings. Nevertheless, in this study, the diagnostic accuracy of the AI algorithms was modest because of small sample size and event number for the training data, and only internal validation was performed. Therefore, to improve the diagnostic performance and objectively accept the utility of our model, further large-scale study with external validation is essential.

In conclusion, our AI models outperformed the current JSCCR guidelines for predicting LNM, suggesting the applicability of AI models in clinical practice to more accurately identify patients who require additional surgery after the initial endoscopic resection of T1 CRC. Further, larger studies are warranted to enhance the performance of AI models for predicting LNM in T1 CRC by developing deep learning models like convolutional neural networks or transformers. Additionally, integrating new AI technologies such as large language models and large multi-modal models with electronic health record systems, high-resolution endoscopic and pathologic image data could improve clinical decision-making regarding the necessity of additional surgery after endoscopic resection of T1 CRC. These advancements in AI technology could significantly impact the future of CRC management.

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (HR20C0026).

No potential conflict of interest relevant to this article was reported.

Study concept and design: S.W.H., N.K., J.S.B. Data acquisition: J.E.B., S.W.H. Data analysis and interpretation: J.E.B., H.Y., S.W.H. Drafting of the manuscript: J.E.B., H.Y., S.W.H., S.S., J.S.B. Critical revision of the manuscript for important intellectual content: J.E.B., H.Y., S.W.H., S.S., J.Y.L., S.W.H., S.H.P., D.H.Y., B.D.Y., S.J.M., S.K.Y., N.K., J.S.B. Statistical analysis: J.E.B., H.Y., S.W.H. Obtained funding: N.K., J.S.B. Administrative, technical, or material support; study supervision: J.E.B., H.Y., S.W.H., S.S., J.Y.L., S.W.H., S.H.P., D.H.Y., B.D.Y., S.J.M., S.K.Y., N.K., J.S.B. Approval of final manuscript: all authors.

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Supplementary materials can be accessed at https://doi.org/10.5009/gnl240273.