Despite the development of surgical technique and perioperative treatment, pancreatic cancer is still a lethal disease worldwide. Although 5-year overall survival (OS) of pancreatic cancer gradually increased in Korea, it was 11.4% in 2016, which was low compared with other gastrointestinal malignancies.¹ However, in patients who underwent pancreatectomy for nonmetastatic pancreatic adenocarcinoma (PDAC), 5-year OS rate was 20.2%, varying according to the 8th edition of the American Joint Committee on Cancer (AJCC) staging of pancreatic cancer, from 38.2% of stage IA to 0.0% of stage III.^2,3 Therefore, the prognosis of resected PDAC differs according to the disease severity, and the accurate prediction of survival probabilities is required to provide the information to the patients about their prognosis and the postoperative individualized treatment plan.

Although the AJCC staging system has been widely used, with many surgeons depending on it for predicting the prognosis, its prognostic accuracy was insufficient with concordance index (C-index) of 0.57 and 0.60 in two large-scaled external validation studies.^3,4 There were some models for predicting prognosis after surgery in patients with resected PDAC,^5-8 consisting of the AJCC T and N stage, along with other variables, such as age, sex, histologic differentiation, adjuvant treatment, or resection margin status. The formulated prediction models had higher performance power than that of the AJCC staging system, and provided the quantitative survival probability after the surgery.^6-8

The aim of this study was to develop a prediction model for providing survival probability for resected PDAC with data from a Surveillance, Epidemiology and End Results (SEER) database in the United States and externally validate this model with nationwide Korean database.

This was a retrospective cohort study with prospectively collected data in the United States and Republic of Korea. The institutional review board of each participating center in Korea approved this study (representative IRB number: 07-2020-058 in Seoul Metropolitan Government Seoul National University Boramae Medical Center, Seoul, Korea) and all patients provided consent to participate in this study.

1. Study population

Fig. 1 is the flowchart of study design for defining the final model development set and test set, including the inclusion and exclusion criteria. For unification of the study period, patients who underwent upfront curative-intent pancreatectomy between 2004 and 2016 were enrolled in this study. Data from a SEER database in National Cancer Institute were utilized for the development of survival prediction models of resected PDAC. The patient’s inclusion criteria were: (1) a confirmed primary site in the pancreas (C250, C251, C252); and (2) ductal adenocarcinoma (SEER histologic code: 8010/3, 8140/3, 8500/3). The detailed inclusion and exclusion codes of SEER are described in Supplementary Table 1. After exclusion, a total of 9,624 patients who underwent pancreatectomy due to PDAC were finally included in the model development set (Fig. 1A). Clinical variables derived from the SEER database were age, sex, histologic differentiation (well differentiated, moderately differentiated, and poorly differentiated), tumor location (head, body, or tail), adjuvant chemotherapy, and the 8th edition of AJCC T and N stage.²

Figure 1. The case selection criteria for defining a model development set and an external validation set.
SEER, Surveillance, Epidemiology, and End Results; KOTUS-BP, Korea Tumor Registry System-Biliary Pancreas.

A Korea Tumor Registry System-Biliary Pancreas (KOTUS-BP) database was established and launched by the Korean Association of Hepato-Biliary-Pancreatic Surgery in 2014. After exclusion, a total of 3,282 patients in 20 institutions in Korea were finally included in the external validation set (Fig. 1B). Clinical variables analyzed were same as the model development set variables.

2. Model development and external validation

Fig. 2 shows the overall scheme of model development and external validation. We exhaustively searched for the best models for each number of variables by comparing the Akaike information criterion (AIC), Harrell C-index, and the 2-year time-dependent area under the receiver operating characteristic curve (AUC) (Step 1). AIC estimates the prediction error and penalized for the number of variables in the model and lower value is better.⁹ C-index calculates the concordance of the predicted and the observed survival time.¹⁰ The closer the value of C-index is to 1, the higher the matching rate. Time-dependent AUC assesses the predictive accuracy of the survival model.¹¹ For survival data, cumulative sensitivity, and dynamic specificity are used to get time-dependent AUC. The closer the value is to 1, the better the predictive ability of the corresponding time cutoff value. For all possible combination of variables, we selected the models with the highest C-index, the highest AUC, or equivalently the lowest AIC for each number of variables. From the models selected from the previous step, we finally determined the single best model using 10-fold cross validation (CV) (Step 2). The best model had the highest 10-fold CV C-index or 10-fold CV 2-year time-dependent AUC.

Figure 2. Flowchart of model development and external validation process.
SEER, Surveillance, Epidemiology and End Results; KOTUS-BP, Korea Tumor Registry System-Biliary Pancreas; AIC, Akaike information criterion; AUC, area under the receiver operating characteristic curve; C-index, Harrell concordance index.

After the determination of the best model, the external validation was performed with the external validation set (KOTUS-BP). C-index along with 1-year, 2-year, 3-year, and 5-year time-dependent AUC were calculated.

3. Statistical analysis

All statistical analyses used R program version 3.6.2 (The R Foundation for Statistical Computing, Vienna, Austria). Survival outcomes were calculated using the Kaplan-Meier method and compared using the log-rank test. Variables for which p-value <0.05 in univariate analysis were entered into a multivariate Cox proportional hazard (PH) model to estimate the hazard ratios (HRs) for the corresponding predictors. Continuous variables were reported as the mean±standard deviation. A calibration plot was used to compare the predicted probability with the observed probability at a specific time point. If the model is ideal, pairs of observed and the predicted probabilities lie on the 45° angle line. To assess calibration for the prognostic model, the Greenwood-Nam-D’Agostino goodness of fit test was performed in each time cutoff value.¹² The nomogram plot was produced using “rms” packages in R program and the calibration plot was produced using “pec” packages.

1. Summary of 5-year OS rates according to the clinical variables in SEER and KOTUS-BP

Table 1 shows the 5-year OS rates according to the clinical variables in SEER and KOTUS-BP database, respectively. Overall, 5-year OS rate in SEER was 20.1% and median survival duration was 21 months. Five-year OS rate in KOTUS-BP was 32.1% and median survival duration was 24 months. Sex, tumor location, AJCC 8th T and N stage, histologic differentiation, and adjuvant chemotherapy could discriminate 5-year OS rates with statistical significance in the univariate analysis in both SEER and KOTUS-BP databases.

Table 1 . Five-Year Overall Survival Rates According to the Variables in the SEER Database and KOTUS-BP.

Variable	SEER database (n=9,624)				KOTUS-BP (n=3,282)
	Patients	5-Year OS, %	p-value*		Patients	5-Year OS, %	p-value*
Age, yr	65.6±10.5	20.1†			63.9±10.1	32.1†
Sex			<0.001				0.007
Female	4,755 (49.4)	21.3			1,381 (42.1)	36.1
Male	4,869 (50.6)	18.9			1,901 (57.9)	29.2
Tumor location			0.002				<0.001
Head	8,079 (83.9)	19.2			2,046 (62.3)	28.4
Body/tail	1,545 (16.1)	25.0			1,236 (37.7)	37.7
AJCC 8th T stage
T1	1,603 (16.7)	32.7			671 (20.5)	45.1
T2	5,830 (60.6)	18.8	<0.001		2,009 (61.2)	29.6	<0.001
T3	2,191 (22.7)	14.3	<0.001		602 (18.3)	24.5	<0.001
AJCC 8th N stage
N0	3,155 (32.8)	32.4			1,312 (40.0)	42.5
N1	4,030 (41.9)	16.8	<0.001		2,363 (72.0)	28.4	<0.001
N2	2,439 (25.3)	9.6	<0.001		543 (16.5)	16.4	<0.001
Differentiation
Well	1,013 (10.5)	37.4			376 (11.5)	44.9
Moderately	5,055 (52.5)	20.5	<0.001		2,363 (72.0)	32.9	<0.001
Poorly	3,556 (37.0)	14.6	<0.001		543 (16.5)	20.5	<0.001
Adjuvant chemotherapy			<0.001				0.004
Yes	2,948 (30.6)	21.3			2,008 (61.2)	36.0
No	6,676 (69.4)	17.3			1,274 (38.8)	29.4

Data are presented as mean±SD or number (%)..

OS, overall survival; SEER, Surveillance, Epidemiology and End Results; KOTUS-BP, Korea Tumor Registry System-Biliary Pancreas; AJCC, American Joint Committee on Cancer..

*Log-rank test..

2. Potential variables for model development in SEER database

In the multivariate Cox-PH model, age (HR, 1.007; 95% confidence interval [CI], 1.004 to 1.009; p<0.001), male sex (HR, 1.070; 95% CI, 1.019 to 1.123; p=0.006), head cancer (HR, 1.12; 95% CI, 1.041 to 1.198; p=0.002), AJCC 8th T stage (T2: HR, 1.394; 95% CI, 1.297 to 1.498; p<0.001 and T3: HR, 1.723; 95% CI, 1.586 to 1.872; p<0.001), N stage (N1: HR, 1.566; 95% CI, 1.476 to 1.663; p<0.001 and N2: HR, 1.980; 95% CI, 1.852 to 2.116; p<0.001), histologic differentiation (moderately differentiated: HR, 1.565; 95% CI, 1.428 to 1.715; p<0.001 and poorly differentiated: HR, 2.069; 95% CI, 1.884 to 2.272; p<0.001), and no adjuvant chemotherapy (HR, 1.789; 95% CI, 1.696 to 1.887; p<0.001) were independent prognostic factors for worse outcome in patients with resected PDAC (Table 2).

Table 2 . Variables for Model Development in the Multivariate Cox Proportional Hazard Model.

Variable	SEER database (n=9,624)
	Hazard ratio	95% CI	p-value
Age	1.007	1.004–1.009	<0.001
Sex
Female	Reference	-	-
Male	1.070	1.019–1.123	0.006
Tumor location
Body/tail	Reference	-	-
Head	1.12	1.041–1.198	0.002
AJCC 8th T stage
T1	Reference	-	-
T2	1.394	1.297–1.498	<0.001
T3	1.723	1.586–1.872	<0.001
AJCC 8th N stage
N0	Reference	-	-
N1	1.566	1.476–1.663	<0.001
N2	1.980	1.852–2.116	<0.001
Differentiation
Well	Reference	-	-
Moderately	1.565	1.428–1.715	<0.001
Poorly	2.069	1.884–2.272	<0.001
Adjuvant chemotherapy
Yes	Reference	-	-
No	1.789	1.696–1.887	<0.001

SEER, Surveillance, Epidemiology and End Results; CI, confidence interval; AJCC, American Joint Committee on Cancer..

3. Model development with SEER database and external validation with KOTUS-BP database

After the exhaustive search, the best combination of variables with the lowest AIC, the highest C-index and the highest 2-year time-dependent AUC were models including all potential variables (Fig. 2, Step 1). For the models with all variables, the single best model was fitted and determined using 10-fold CV. The C-index, 1-year, 2-year, and 3-year time-dependent AUCs of the best model after the 10-fold CV were 0.654, 0.712, 0.689, and 0.694, respectively (Fig. 2, Step 2). This model was visualized to nomogram form (Fig. 3).

Figure 3. Nomogram for survival in patients with resected pancreatic ductal adenocarcinoma. Prediction of survival can be made by drawing a vertical line from the total points scale to the survival probabilities scale.
M, male; F, female; WD, well differentiated; MD, moderately differentiated; PD, poorly differentiated; AJCC, American Joint Committee on Cancer.

Then, the external validation was performed with KOTUS-BP database (Fig. 2, Step 3). The C-index, 1-year, 2-year, 3-year, and 5-year time-dependent AUCs were 0.628, 0.650, 0.665, 0.675, and 0.686, respectively (see Supplementary Fig. 1). The p-value of Greenwood-Nam-D’Agostino test for each time cutoff value (1-, 2-, 3-, and 5-year) was 0.78, 0.18, 0.39, and 0.17, respectively, indicating the developed model to be well calibrated (Fig. 4).

Figure 4. Calibration curves for 1-year, 2-year, 3-year and 5-year overall survival.

This study conducted the model development with SEER database by the rigorous statistical techniques and demonstrated the reliable performance (C-index of 0.628) in the external validation with nationwide database in Korea (KOTUS-BP). Age, sex, histologic differentiation, AJCC 8th T and N stage, tumor location, and adjuvant chemotherapy were included in this prediction model, with a few of them already included in other prognostic models.^6-8 The visualized nomogram was established, and time-dependent survival probability could be easily calculated (Fig. 3). Calibration demonstrated good consistency between the predictive survival and actual observation of 1-year, 2-year, 3-year, and 5-year survival rates (Fig. 4).

The development of nomogram is already popular among clinicians for a variety of diseases as they provide quantitative information by simple calculation, helping clinicians decide the customized treatment strategies.^13-17 The current AJCC tumor-node-metastasis staging system discriminate patients with limited variables, and the C-index of 8th edition of AJCC pancreatic cancer stage was 0.588, which was lower than that of the present model (0.628). However, variables other than tumor size or lymph nodes, such as perineural invasion,¹⁸ resection margin status,¹⁹ or adjuvant chemotherapy,²⁰ are also associated with survival outcomes in patients with pancreatic cancer. Altogether, three studies developed the nomograms for predicting the survival of resected nonmetastatic PDAC and revealed higher performances than that of the 8th edition of AJCC staging system.^6-8 For the customized treatment, the current tumor-node-metastasis staging system should be revised or replaced with more delicate and accurate prediction model.

Different from the other prediction models,^7,8 this model included the tumor location. Traditionally, the OS rate of pancreatic body or tail cancer was lower than that of the pancreatic head cancer, because of its discovery at more advanced or distant metastatic status, and the lower R0 resection rate.^21-23 However, in this prediction model, tumor location was associated with survival outcome in resected PDAC with the head cancer showing worse survival outcome than body or tail cancer (HR, 1.12; 95% CI, 1.041 to 1.198; p=0.002). It was probably because this prediction model was developed only with the patients who underwent curative-intent pancreatectomy. The patients with body or tail lesion showed better survival outcome than those with head lesion in resected PDAC in two studies. In the Nationwide Inpatient Sample database in the United States, patients who underwent distal pancreatectomy showed lower in-hospital mortality rate than those who underwent pancreatoduodenectomy.²⁴ The nationwide database in Germany showed similar results with the lower mortality rate of distal pancreatectomy than that of pancreatoduodenectomy in patients with pancreatic neoplasms.²⁵ It might be due to pancreatoduodenectomy being more invasive than distal pancreatectomy, with pancreatoduodenectomy showing higher mortality rate in patients requiring intervention for the pancreatoduodenectomy-related complications.²⁵

Adjuvant chemotherapy for resected PDAC was considered to be the standard treatment because survival outcome was better in patients with adjuvant chemotherapy than in patients with observation after pancreatectomy.²⁶ In this study, the patients who received adjuvant chemotherapy had better 5-year OS rate than those who did not, in SEER database (21.3% vs 17.3%, p<0.001) (Table 1) and KOTUS-BP (36.0% vs 29.4%, p=0.004) (Table 1). In the multivariate Cox-PH model, no adjuvant chemotherapy was associated with worse survival outcome (HR, 1.789; 95% CI, 1.696 to 1.887; p<0.001) (Table 2). Although the detailed chemotherapy regimens were not investigated in this study, and the chemotherapy regimens might be heterogeneous in both databases, this model suggested that with or without adjuvant chemotherapy had more statistical power than that of AJCC T stage because of higher HR (1.789) of no chemotherapy than that of AJCC T stage (HR T2 vs T1, 1.394; HR T3 vs T1, 1.723).

Histologic differentiation was one of the prognostic factors associated with poor survival outcome in other malignancies.^27,28 A study revealed the association of histologic differentiation with survival outcome of PDAC in the univariate analysis.²⁹ In this study, the histologic differentiation discriminated the survival outcomes with statistical significance in both databases, with poor survival outcome in the multivariate Cox-PH model (Table 2). In addition, the statistical power of histologic differentiation was similar to that of AJCC N stage with comparable HRs comparable between two variables.

Because the nomograms were established based on the prognostic factors related with the disease, similar variables would be selected and included among the models of the same disease. Although previous models of PDAC were developed with different cohorts, the variables included in these models were quite similar (e.g., AJCC T stage, N stage or lymph node ratio, histologic differentiation, resection margin status, etc.).^5-8 In previous survival prediction models, their performances were also comparable with the C-indices of these models in the external validations as 0.58 to 0.65 (Table 3).^6-8 Consequently, the predictive performance of models consisting of clinical variables only using conventional multivariate Cox-PH analysis seems difficult to exceed 0.7. Recently, machine learning techniques have been utilized to develop prediction models for increasing the prediction power better than the conventional multivariate logistic regression model.³⁰ In addition, if high-dimensional variables, such as genomics or transcriptomics data, were available, the performance might be improved.³¹ Therefore, a better prediction model might be established when the clinical information data along with genomic data, and other statistical methods are utilized.

Table 3 . Summary of Previous Studies That Had Independent Model Development and External Validation.

Study	Model development cohort	External validation cohort	C-index
Huang et al.6	SEER database (n=9,519)	European database (4 countries, n=2,318)	0.58–0.63
Pu et al.7	SEER database (n=12,343)	Zhongshan Hospital (n=127)	0.63
van Roessel et al.8	International database (8 countries, n=3,081)	Academic Medical Center, Amsterdam (n=350)	0.65

SEER, Surveillance, Epidemiology and End Results; C-index, Harrell concordance index..

This study had some limitations being a retrospective study. In addition, because the two databases were national cohort-based, specific chemotherapy regimens were not investigated, thereby not reflecting the effect of survival benefit of the recent chemotherapy protocols, such as FOLFIRINOX or gemcitabine plus nab-paclitaxel, etc. However, the prediction model in this study was created with the SEER database, one of the largest, qualified database, and was externally validated with the KOTUS-BP data, prospectively registered and regularly managed by the pancreatobiliary surgeons at the specialized centers in Korea. The lack of detailed regimen of adjuvant chemotherapy without information about resection margin status and information on whether neoadjuvant chemotherapy was performed or not in the SEER database were other limitations. The difference of race distribution between two databases was also one of the limitations that 81.6% of patients were White, 10.4% were Black, and less than 5% were Eastern-Northern Asian (1.6% of Chinese, 1.2% of Japanese, 0.96% of Korean) in the SEER database. The international collaborate prospective studies should be performed in future, to develop and validate the global prediction model of resected PDAC with higher performance power. Moreover, a new prediction model with preoperative variables would be helpful for clinicians to decide tailored treatment strategy for treating pancreatic cancer.

In conclusion, the survival prediction model of resected PDAC could predict the 1-, 2-, 3-, and 5-year survival with the reliable performance (C-indices, 0.650, 0.665, 0.675, and 0.686, respectively) when applied to the Korean patients. The external validation studies with other nationwide databases are needed to evaluate the performance power of this model.

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute KHIDI, funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI16C2307) and the Collaborative Genome Program for Fostering New Post-Genome Industry of the National Research Foundation funded by the Ministry of Science and ICT (NRF-2017M3C9A5031591).

See editorial on page 797.

No potential conflict of interest relevant to this article was reported.

Study concept and design: J.S.K., L.M., J.S.H., C.S.L., J.Y.J., T.P. Data acquisition, analysis and interpretation: J.S.K., I.W.H., S.H.S., Y.S.Y., H.S.H., D.W.H., J.H.L., W.J.L., S.J.P., J.S.P., Y.K., H.L., Y.D.Y., J.D.Y., S.E.L., I.Y.P., C.Y.J., Y.R, S.R.K., J.I.M., S.K.L., H.J.K., H.K., W.K. Drafting of the manuscript, critical revision: J.S.K., L.M., C.S.L., J.Y.J. Statistical analysis: J.S.K., L.M., S.L., T.P. Obtained funding: J.Y.J., T.P. Study supervision: C.S.L., J.Y.J., T.P.