2019-01-31 11:07:41: xmlchange success ./xmlchange DOUT_S=TRUE --------------------------------------------------- 2019-01-31 11:07:41: xmlchange success ./xmlchange RESUBMIT=17 --------------------------------------------------- 2019-01-31 11:07:41: xmlchange success ./xmlchange STOP_N=2 --------------------------------------------------- 2019-01-31 11:07:42: xmlchange success ./xmlchange STOP_OPTION=nyears --------------------------------------------------- 2019-01-31 11:07:47: case.setup starting --------------------------------------------------- 2019-01-31 11:07:48: case.setup success --------------------------------------------------- 2019-01-31 11:08:07: xmlchange success ./xmlchange RESUBMIT=0 --------------------------------------------------- 2019-01-31 15:39:17: xmlchange success ./xmlchange SSTICE_DATA_FILENAME=/glade/work/brianpm/model_data/CFMIP/cfmip_sst_p4k_HadOIBl_bc_0.9x1.25_1850_2017_c20190129.nc --------------------------------------------------- 2019-01-31 15:39:46: case.setup starting --------------------------------------------------- 2019-01-31 15:39:47: case.setup success --------------------------------------------------- 2019-01-31 15:45:57: case.build starting --------------------------------------------------- 2019-01-31 15:45:58: case.build error ERROR: Command /gpfs/u/home/cmip6/cesm_tags/cesm2.1-exp007/components/cam/bld/build-namelist -ntasks 360 -csmdata /glade/p/cesmdata/cseg/inputdata -infile /glade/work/cmip6/cases/CFMIP/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/Buildconf/camconf/namelist -ignore_ic_year -use_case hist_cam6 -inputdata /glade/work/cmip6/cases/CFMIP/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/Buildconf/cam.input_data_list -namelist " &atmexp /" failed rc=255 out= err=::ERROR(Build::Namelist::_parse_next): expect a F90 constant for a namelist instead got: fincl4 = --------------------------------------------------- 2019-01-31 15:46:50: case.build starting --------------------------------------------------- 2019-01-31 15:46:51: case.build error ERROR: Command /gpfs/u/home/cmip6/cesm_tags/cesm2.1-exp007/components/clm/bld/build-namelist failed rc=255 out= err=::ERROR(Build::Namelist::_parse_next): expect a F90 constant for a namelist instead got: hist_fincl3 = --------------------------------------------------- 2019-01-31 15:47:50: case.build starting --------------------------------------------------- CESM version is cesm2.1-exp007 Processing externals description file : Externals.cfg Processing externals description file : Externals_CLM.cfg Processing externals description file : Externals_POP.cfg Processing externals description file : Externals_CISM.cfg Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, M ./cime modified sandbox, on cime_cesm2_1_rel_02 ./components/cam clean sandbox, on cam1/trunk_tags/cam6_0_026/components/cam ./components/cice clean sandbox, on cice5_20181109 ./components/cism clean sandbox, on release-cesm2.0.04 ./components/cism/source_cism clean sandbox, on release-cism2.1.03 ./components/clm clean sandbox, on release-clm5.0.08 ./components/clm/src/fates clean sandbox, on fates_s1.8.1_a3.0.0 ./components/clm/tools/PTCLM clean sandbox, on PTCLM2_180611 ./components/mosart clean sandbox, on release-cesm2.0.01 ./components/pop clean sandbox, on pop2/trunk_tags/cesm_pop_2_1_20181004 ./components/pop/externals/CVMix clean sandbox, on v0.90-beta ./components/pop/externals/MARBL clean sandbox, on marbl0.32.0 ./components/rtm clean sandbox, on release-cesm2.0.00 ./components/ww3 clean sandbox, on ww3/trunk_tags/ww3_181001 2019-01-31 15:59:09: case.build success --------------------------------------------------- 2019-01-31 17:04:50: case.run starting --------------------------------------------------- 2019-01-31 17:04:55: model execution starting --------------------------------------------------- 2019-01-31 17:05:21: model execution success --------------------------------------------------- 2019-01-31 17:05:21: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4205682.chadmin1.190131-170450 --------------------------------------------------- 2019-02-01 11:00:44: case.setup starting --------------------------------------------------- 2019-02-01 11:00:45: case.setup success --------------------------------------------------- 2019-02-01 11:00:54: case.setup starting --------------------------------------------------- 2019-02-01 11:00:55: case.setup success --------------------------------------------------- 2019-02-01 11:01:57: build.clean starting --------------------------------------------------- 2019-02-01 11:02:05: build.clean success --------------------------------------------------- 2019-02-01 11:10:41: xmlchange success ./xmlchange CAM_CONFIG_OPTS=-cosp --append --------------------------------------------------- 2019-02-01 11:12:54: xmlchange success ./xmlchange --subgroup case.run JOB_WALLCLOCK_TIME=6:00:00 --------------------------------------------------- 2019-02-01 11:13:28: case.setup starting --------------------------------------------------- 2019-02-01 11:13:29: case.setup success --------------------------------------------------- 2019-02-01 13:29:16: case.build starting --------------------------------------------------- CESM version is cesm2.1-exp007 Processing externals description file : Externals.cfg Processing externals description file : Externals_CLM.cfg Processing externals description file : Externals_POP.cfg Processing externals description file : Externals_CISM.cfg Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, M ./cime modified sandbox, on cime_cesm2_1_rel_02 ./components/cam clean sandbox, on cam1/trunk_tags/cam6_0_026/components/cam ./components/cice clean sandbox, on cice5_20181109 ./components/cism clean sandbox, on release-cesm2.0.04 ./components/cism/source_cism clean sandbox, on release-cism2.1.03 ./components/clm clean sandbox, on release-clm5.0.08 ./components/clm/src/fates clean sandbox, on fates_s1.8.1_a3.0.0 ./components/clm/tools/PTCLM clean sandbox, on PTCLM2_180611 ./components/mosart clean sandbox, on release-cesm2.0.01 ./components/pop clean sandbox, on pop2/trunk_tags/cesm_pop_2_1_20181004 ./components/pop/externals/CVMix clean sandbox, on v0.90-beta ./components/pop/externals/MARBL clean sandbox, on marbl0.32.0 ./components/rtm clean sandbox, on release-cesm2.0.00 ./components/ww3 clean sandbox, on ww3/trunk_tags/ww3_181001 2019-02-01 13:40:12: case.build success --------------------------------------------------- 2019-02-01 13:46:00: case.run starting --------------------------------------------------- 2019-02-01 13:46:04: model execution starting --------------------------------------------------- 2019-02-01 13:46:33: model execution success --------------------------------------------------- 2019-02-01 13:46:33: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4212701.chadmin1.190201-134600 --------------------------------------------------- 2019-02-01 13:51:09: case.run starting --------------------------------------------------- 2019-02-01 13:51:14: model execution starting --------------------------------------------------- 2019-02-01 13:51:42: model execution success --------------------------------------------------- 2019-02-01 13:51:42: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4212712.chadmin1.190201-135109 --------------------------------------------------- 2019-02-01 14:06:31: case.run starting --------------------------------------------------- 2019-02-01 14:06:36: model execution starting --------------------------------------------------- 2019-02-01 14:10:10: model execution success --------------------------------------------------- 2019-02-01 14:10:10: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4212911.chadmin1.190201-140631 --------------------------------------------------- 2019-02-01 14:27:43: case.run starting --------------------------------------------------- 2019-02-01 14:27:48: model execution starting --------------------------------------------------- 2019-02-01 14:31:37: model execution success --------------------------------------------------- 2019-02-01 14:31:37: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4212943.chadmin1.190201-142743 --------------------------------------------------- 2019-02-01 14:42:53: case.run starting --------------------------------------------------- 2019-02-01 14:42:58: model execution starting --------------------------------------------------- 2019-02-01 14:46:49: model execution success --------------------------------------------------- 2019-02-01 14:46:49: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4213506.chadmin1.190201-144253 --------------------------------------------------- 2019-02-01 14:55:23: case.run starting --------------------------------------------------- 2019-02-01 14:55:28: model execution starting --------------------------------------------------- 2019-02-01 14:59:10: model execution success --------------------------------------------------- 2019-02-01 14:59:10: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 360 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4213543.chadmin1.190201-145523 --------------------------------------------------- 2019-02-01 15:02:42: case.run starting --------------------------------------------------- 2019-02-01 15:02:47: model execution starting --------------------------------------------------- 2019-02-06 09:07:19: case.run starting --------------------------------------------------- 2019-02-06 09:07:24: model execution starting --------------------------------------------------- 2019-02-08 12:25:42: xmlchange success ./xmlchange STOP_N=5 --------------------------------------------------- 2019-02-08 12:25:50: xmlchange success ./xmlchange STOP_OPTION=ndays --------------------------------------------------- 2019-02-08 12:27:38: case.run starting --------------------------------------------------- 2019-02-08 12:27:43: model execution starting --------------------------------------------------- 2019-02-08 12:33:49: model execution success --------------------------------------------------- 2019-02-08 12:33:49: case.run success --------------------------------------------------- 2019-02-08 12:33:49: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-08 12:34:00: st_archive starting --------------------------------------------------- 2019-02-08 12:34:12: st_archive success --------------------------------------------------- 2019-02-08 13:58:46: setup.clean starting --------------------------------------------------- 2019-02-08 13:58:46: setup.clean success --------------------------------------------------- 2019-02-08 13:58:51: case.setup starting --------------------------------------------------- 2019-02-08 13:58:52: build.clean starting --------------------------------------------------- 2019-02-08 13:59:09: build.clean success --------------------------------------------------- 2019-02-08 13:59:10: case.setup success --------------------------------------------------- 2019-02-08 13:59:51: build.clean starting --------------------------------------------------- 2019-02-08 14:02:46: build.clean success --------------------------------------------------- 2019-02-08 14:04:20: case.build starting --------------------------------------------------- 2019-02-08 14:08:41: xmlchange success ./xmlchange RESUBMIT=0 --------------------------------------------------- CESM version is cesm2.1-exp007 Processing externals description file : Externals.cfg Processing externals description file : Externals_CLM.cfg Processing externals description file : Externals_POP.cfg Processing externals description file : Externals_CISM.cfg Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, M ./cime modified sandbox, on cime_cesm2_1_rel_02 ./components/cam clean sandbox, on cam1/trunk_tags/cam6_0_026/components/cam ./components/cice clean sandbox, on cice5_20181109 ./components/cism clean sandbox, on release-cesm2.0.04 ./components/cism/source_cism clean sandbox, on release-cism2.1.03 ./components/clm clean sandbox, on release-clm5.0.08 ./components/clm/src/fates clean sandbox, on fates_s1.8.1_a3.0.0 ./components/clm/tools/PTCLM clean sandbox, on PTCLM2_180611 ./components/mosart clean sandbox, on release-cesm2.0.01 ./components/pop clean sandbox, on pop2/trunk_tags/cesm_pop_2_1_20181004 ./components/pop/externals/CVMix clean sandbox, on v0.90-beta ./components/pop/externals/MARBL clean sandbox, on marbl0.32.0 ./components/rtm clean sandbox, on release-cesm2.0.00 ./components/ww3 clean sandbox, on ww3/trunk_tags/ww3_181001 2019-02-08 14:09:04: case.build success --------------------------------------------------- 2019-02-08 15:28:57: case.run starting --------------------------------------------------- 2019-02-08 15:29:01: model execution starting --------------------------------------------------- 2019-02-08 17:09:59: model execution success --------------------------------------------------- 2019-02-08 17:09:59: case.run success --------------------------------------------------- 2019-02-08 17:10:01: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-08 17:10:14: st_archive starting --------------------------------------------------- 2019-02-08 17:11:00: st_archive success --------------------------------------------------- 2019-02-08 17:29:01: case.run starting --------------------------------------------------- 2019-02-08 17:29:06: model execution starting --------------------------------------------------- 2019-02-08 19:02:52: model execution success --------------------------------------------------- 2019-02-08 19:02:52: case.run success --------------------------------------------------- 2019-02-08 19:02:54: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-08 19:03:09: st_archive starting --------------------------------------------------- 2019-02-08 19:03:24: st_archive success --------------------------------------------------- 2019-02-08 19:17:36: case.run starting --------------------------------------------------- 2019-02-08 19:17:40: model execution starting --------------------------------------------------- 2019-02-08 20:51:43: model execution success --------------------------------------------------- 2019-02-08 20:51:43: case.run success --------------------------------------------------- 2019-02-08 20:51:44: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-08 20:51:59: st_archive starting --------------------------------------------------- 2019-02-08 20:52:14: st_archive success --------------------------------------------------- 2019-02-08 21:05:59: case.run starting --------------------------------------------------- 2019-02-08 21:06:04: model execution starting --------------------------------------------------- 2019-02-08 22:39:09: model execution success --------------------------------------------------- 2019-02-08 22:39:09: case.run success --------------------------------------------------- 2019-02-08 22:39:11: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-08 22:39:23: st_archive starting --------------------------------------------------- 2019-02-08 22:39:38: st_archive success --------------------------------------------------- 2019-02-08 22:48:10: case.run starting --------------------------------------------------- 2019-02-08 22:48:15: model execution starting --------------------------------------------------- 2019-02-09 00:21:20: model execution success --------------------------------------------------- 2019-02-09 00:21:20: case.run success --------------------------------------------------- 2019-02-09 00:21:21: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 00:21:32: st_archive starting --------------------------------------------------- 2019-02-09 00:21:47: st_archive success --------------------------------------------------- 2019-02-09 00:21:57: case.run starting --------------------------------------------------- 2019-02-09 00:22:01: model execution starting --------------------------------------------------- 2019-02-09 01:55:09: model execution success --------------------------------------------------- 2019-02-09 01:55:09: case.run success --------------------------------------------------- 2019-02-09 01:55:10: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 01:55:22: st_archive starting --------------------------------------------------- 2019-02-09 01:55:38: st_archive success --------------------------------------------------- 2019-02-09 02:02:38: case.run starting --------------------------------------------------- 2019-02-09 02:02:43: model execution starting --------------------------------------------------- 2019-02-09 03:39:24: model execution success --------------------------------------------------- 2019-02-09 03:39:24: case.run success --------------------------------------------------- 2019-02-09 03:39:26: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 04:25:30: st_archive starting --------------------------------------------------- 2019-02-09 04:25:43: st_archive success --------------------------------------------------- 2019-02-09 04:32:28: case.run starting --------------------------------------------------- 2019-02-09 04:32:33: model execution starting --------------------------------------------------- 2019-02-09 06:05:52: model execution success --------------------------------------------------- 2019-02-09 06:05:52: case.run success --------------------------------------------------- 2019-02-09 06:05:53: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 06:06:04: st_archive starting --------------------------------------------------- 2019-02-09 06:06:17: st_archive success --------------------------------------------------- 2019-02-09 06:06:26: case.run starting --------------------------------------------------- 2019-02-09 06:06:29: model execution starting --------------------------------------------------- 2019-02-09 07:39:56: model execution success --------------------------------------------------- 2019-02-09 07:39:56: case.run success --------------------------------------------------- 2019-02-09 07:39:58: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 07:40:11: st_archive starting --------------------------------------------------- 2019-02-09 07:40:25: st_archive success --------------------------------------------------- 2019-02-09 07:40:37: case.run starting --------------------------------------------------- 2019-02-09 07:40:42: model execution starting --------------------------------------------------- 2019-02-09 09:15:08: model execution success --------------------------------------------------- 2019-02-09 09:15:08: case.run success --------------------------------------------------- 2019-02-09 09:15:10: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 09:15:20: st_archive starting --------------------------------------------------- 2019-02-09 09:15:35: st_archive success --------------------------------------------------- 2019-02-09 09:15:43: case.run starting --------------------------------------------------- 2019-02-09 09:15:48: model execution starting --------------------------------------------------- 2019-02-09 10:47:13: model execution success --------------------------------------------------- 2019-02-09 10:47:13: case.run success --------------------------------------------------- 2019-02-09 10:47:14: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 10:47:25: st_archive starting --------------------------------------------------- 2019-02-09 10:47:39: st_archive success --------------------------------------------------- 2019-02-09 10:47:48: case.run starting --------------------------------------------------- 2019-02-09 10:47:52: model execution starting --------------------------------------------------- 2019-02-09 12:18:46: model execution success --------------------------------------------------- 2019-02-09 12:18:46: case.run success --------------------------------------------------- 2019-02-09 12:18:47: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 12:18:58: st_archive starting --------------------------------------------------- 2019-02-09 12:19:13: st_archive success --------------------------------------------------- 2019-02-09 12:19:23: case.run starting --------------------------------------------------- 2019-02-09 12:19:28: model execution starting --------------------------------------------------- 2019-02-09 13:52:40: model execution success --------------------------------------------------- 2019-02-09 13:52:40: case.run success --------------------------------------------------- 2019-02-09 13:52:42: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 13:52:53: st_archive starting --------------------------------------------------- 2019-02-09 13:53:08: st_archive success --------------------------------------------------- 2019-02-09 13:53:18: case.run starting --------------------------------------------------- 2019-02-09 13:53:22: model execution starting --------------------------------------------------- 2019-02-09 15:26:09: model execution success --------------------------------------------------- 2019-02-09 15:26:09: case.run success --------------------------------------------------- 2019-02-09 15:26:11: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 15:26:21: st_archive starting --------------------------------------------------- 2019-02-09 15:26:35: st_archive success --------------------------------------------------- 2019-02-09 15:26:44: case.run starting --------------------------------------------------- 2019-02-09 15:26:49: model execution starting --------------------------------------------------- 2019-02-09 17:00:34: model execution success --------------------------------------------------- 2019-02-09 17:00:34: case.run success --------------------------------------------------- 2019-02-09 17:00:35: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 17:00:46: st_archive starting --------------------------------------------------- 2019-02-09 17:01:00: st_archive success --------------------------------------------------- 2019-02-09 17:01:09: case.run starting --------------------------------------------------- 2019-02-09 17:01:12: model execution starting --------------------------------------------------- 2019-02-09 18:36:33: model execution success --------------------------------------------------- 2019-02-09 18:36:33: case.run success --------------------------------------------------- 2019-02-09 18:36:35: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 18:36:46: st_archive starting --------------------------------------------------- 2019-02-09 18:36:59: st_archive success --------------------------------------------------- 2019-02-09 18:37:08: case.run starting --------------------------------------------------- 2019-02-09 18:37:13: model execution starting --------------------------------------------------- 2019-02-09 20:09:38: model execution success --------------------------------------------------- 2019-02-09 20:09:38: case.run success --------------------------------------------------- 2019-02-09 20:09:39: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 20:09:50: st_archive starting --------------------------------------------------- 2019-02-09 20:10:03: st_archive success --------------------------------------------------- 2019-02-09 20:10:13: case.run starting --------------------------------------------------- 2019-02-09 20:10:16: model execution starting --------------------------------------------------- 2019-02-09 21:44:51: model execution success --------------------------------------------------- 2019-02-09 21:44:51: case.run success --------------------------------------------------- 2019-02-09 21:44:52: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 21:45:02: st_archive starting --------------------------------------------------- 2019-02-09 21:45:15: st_archive success --------------------------------------------------- 2019-02-09 21:45:24: case.run starting --------------------------------------------------- 2019-02-09 21:45:28: model execution starting --------------------------------------------------- 2019-02-09 23:18:33: model execution success --------------------------------------------------- 2019-02-09 23:18:33: case.run success --------------------------------------------------- 2019-02-09 23:18:34: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-09 23:18:44: st_archive starting --------------------------------------------------- 2019-02-09 23:18:57: st_archive success --------------------------------------------------- 2019-02-09 23:19:08: case.run starting --------------------------------------------------- 2019-02-09 23:19:12: model execution starting --------------------------------------------------- 2019-02-10 00:52:00: model execution success --------------------------------------------------- 2019-02-10 00:52:00: case.run success --------------------------------------------------- 2019-02-10 00:52:02: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 00:52:13: st_archive starting --------------------------------------------------- 2019-02-10 00:52:26: st_archive success --------------------------------------------------- 2019-02-10 00:52:35: case.run starting --------------------------------------------------- 2019-02-10 00:52:40: model execution starting --------------------------------------------------- 2019-02-10 02:26:13: model execution success --------------------------------------------------- 2019-02-10 02:26:13: case.run success --------------------------------------------------- 2019-02-10 02:26:15: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 02:26:24: st_archive starting --------------------------------------------------- 2019-02-10 02:26:36: st_archive success --------------------------------------------------- 2019-02-10 02:26:46: case.run starting --------------------------------------------------- 2019-02-10 02:26:49: model execution starting --------------------------------------------------- 2019-02-10 03:58:39: model execution success --------------------------------------------------- 2019-02-10 03:58:39: case.run success --------------------------------------------------- 2019-02-10 03:58:40: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 03:58:51: st_archive starting --------------------------------------------------- 2019-02-10 03:59:03: st_archive success --------------------------------------------------- 2019-02-10 03:59:12: case.run starting --------------------------------------------------- 2019-02-10 03:59:15: model execution starting --------------------------------------------------- 2019-02-10 05:31:30: model execution success --------------------------------------------------- 2019-02-10 05:31:30: case.run success --------------------------------------------------- 2019-02-10 05:31:32: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 05:31:42: st_archive starting --------------------------------------------------- 2019-02-10 05:31:57: st_archive success --------------------------------------------------- 2019-02-10 05:32:08: case.run starting --------------------------------------------------- 2019-02-10 05:32:12: model execution starting --------------------------------------------------- 2019-02-10 07:04:22: model execution success --------------------------------------------------- 2019-02-10 07:04:22: case.run success --------------------------------------------------- 2019-02-10 07:04:24: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 07:04:33: st_archive starting --------------------------------------------------- 2019-02-10 07:04:48: st_archive success --------------------------------------------------- 2019-02-10 07:04:57: case.run starting --------------------------------------------------- 2019-02-10 07:05:00: model execution starting --------------------------------------------------- 2019-02-10 08:37:30: model execution success --------------------------------------------------- 2019-02-10 08:37:30: case.run success --------------------------------------------------- 2019-02-10 08:37:31: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 08:37:43: st_archive starting --------------------------------------------------- 2019-02-10 08:37:58: st_archive success --------------------------------------------------- 2019-02-10 08:38:08: case.run starting --------------------------------------------------- 2019-02-10 08:38:12: model execution starting --------------------------------------------------- 2019-02-10 10:11:58: model execution success --------------------------------------------------- 2019-02-10 10:11:58: case.run success --------------------------------------------------- 2019-02-10 10:11:59: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 10:12:11: st_archive starting --------------------------------------------------- 2019-02-10 10:12:26: st_archive success --------------------------------------------------- 2019-02-10 10:12:35: case.run starting --------------------------------------------------- 2019-02-10 10:12:39: model execution starting --------------------------------------------------- 2019-02-10 11:48:42: model execution success --------------------------------------------------- 2019-02-10 11:48:42: case.run success --------------------------------------------------- 2019-02-10 11:48:44: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 11:48:54: st_archive starting --------------------------------------------------- 2019-02-10 11:49:09: st_archive success --------------------------------------------------- 2019-02-10 11:49:19: case.run starting --------------------------------------------------- 2019-02-10 11:49:23: model execution starting --------------------------------------------------- 2019-02-10 13:23:10: model execution success --------------------------------------------------- 2019-02-10 13:23:10: case.run success --------------------------------------------------- 2019-02-10 13:23:11: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 13:23:22: st_archive starting --------------------------------------------------- 2019-02-10 13:23:38: st_archive success --------------------------------------------------- 2019-02-10 13:23:48: case.run starting --------------------------------------------------- 2019-02-10 13:23:51: model execution starting --------------------------------------------------- 2019-02-10 14:58:16: model execution success --------------------------------------------------- 2019-02-10 14:58:16: case.run success --------------------------------------------------- 2019-02-10 14:58:17: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 14:58:29: st_archive starting --------------------------------------------------- 2019-02-10 14:58:43: st_archive success --------------------------------------------------- 2019-02-10 14:58:53: case.run starting --------------------------------------------------- 2019-02-10 14:58:57: model execution starting --------------------------------------------------- 2019-02-10 16:34:30: model execution success --------------------------------------------------- 2019-02-10 16:34:30: case.run success --------------------------------------------------- 2019-02-10 16:34:31: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 16:34:44: st_archive starting --------------------------------------------------- 2019-02-10 16:34:59: st_archive success --------------------------------------------------- 2019-02-10 16:35:10: case.run starting --------------------------------------------------- 2019-02-10 16:35:14: model execution starting --------------------------------------------------- 2019-02-10 18:09:34: model execution success --------------------------------------------------- 2019-02-10 18:09:34: case.run success --------------------------------------------------- 2019-02-10 18:09:35: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 18:09:46: st_archive starting --------------------------------------------------- 2019-02-10 18:10:01: st_archive success --------------------------------------------------- 2019-02-10 18:10:12: case.run starting --------------------------------------------------- 2019-02-10 18:10:16: model execution starting --------------------------------------------------- 2019-02-10 19:43:56: model execution success --------------------------------------------------- 2019-02-10 19:43:56: case.run success --------------------------------------------------- 2019-02-10 19:43:58: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 19:44:09: st_archive starting --------------------------------------------------- 2019-02-10 19:44:26: st_archive success --------------------------------------------------- 2019-02-10 19:52:27: case.run starting --------------------------------------------------- 2019-02-10 19:52:32: model execution starting --------------------------------------------------- 2019-02-10 21:27:47: model execution success --------------------------------------------------- 2019-02-10 21:27:47: case.run success --------------------------------------------------- 2019-02-10 21:27:48: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-10 21:27:59: st_archive starting --------------------------------------------------- 2019-02-10 21:28:14: st_archive success --------------------------------------------------- 2019-02-10 21:28:24: case.run starting --------------------------------------------------- 2019-02-10 21:28:27: model execution starting --------------------------------------------------- 2019-02-10 23:06:30: model execution success --------------------------------------------------- 2019-02-10 23:06:30: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 576 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4326064.chadmin1.190210-212824 --------------------------------------------------- 2019-02-10 23:07:09: case.run starting --------------------------------------------------- 2019-02-10 23:07:14: model execution starting --------------------------------------------------- 2019-02-11 00:41:57: model execution success --------------------------------------------------- 2019-02-11 00:41:57: case.run success --------------------------------------------------- 2019-02-11 00:41:58: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-02-11 00:42:09: st_archive starting --------------------------------------------------- 2019-02-11 00:42:25: st_archive success --------------------------------------------------- 2019-02-11 00:44:46: case.run starting --------------------------------------------------- 2019-02-11 00:44:51: model execution starting --------------------------------------------------- 2019-02-11 02:22:27: model execution success --------------------------------------------------- 2019-02-11 02:22:27: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 576 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4326568.chadmin1.190211-004446 --------------------------------------------------- 2019-02-11 02:23:09: case.run starting --------------------------------------------------- 2019-02-11 02:23:14: model execution starting --------------------------------------------------- 2019-02-11 02:24:50: model execution success --------------------------------------------------- 2019-02-11 02:24:50: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 576 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4326688.chadmin1.190211-022309 --------------------------------------------------- 2019-02-11 02:26:59: case.run starting --------------------------------------------------- 2019-02-11 02:27:03: model execution starting --------------------------------------------------- 2019-02-11 02:28:35: model execution success --------------------------------------------------- 2019-02-11 02:28:35: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 576 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4326691.chadmin1.190211-022659 --------------------------------------------------- 2019-02-11 02:38:43: case.run starting --------------------------------------------------- 2019-02-11 02:38:48: model execution starting --------------------------------------------------- 2019-02-11 02:40:22: model execution success --------------------------------------------------- 2019-02-11 02:40:22: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -np 576 -p "%g:" omplace -tm open64 /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/f.e21.FHIST_BGC.f09_f09_mg17.CFMIP-amip-p4k.001/run/cesm.log.4326713.chadmin1.190211-023843 ---------------------------------------------------