2019-05-02 16:28:04: xmlchange success ./xmlchange DOUT_S=TRUE --------------------------------------------------- 2019-05-02 16:28:04: xmlchange success ./xmlchange RESUBMIT=7 --------------------------------------------------- 2019-05-02 16:28:04: xmlchange success ./xmlchange STOP_N=10 --------------------------------------------------- 2019-05-02 16:28:05: xmlchange success ./xmlchange STOP_OPTION=nyears --------------------------------------------------- 2019-05-02 16:28:11: case.setup starting --------------------------------------------------- 2019-05-02 16:28:13: case.setup success --------------------------------------------------- 2019-05-02 16:28:43: xmlchange success ./xmlchange RESUBMIT=0 --------------------------------------------------- 2019-05-02 16:47:18: setup.clean starting --------------------------------------------------- 2019-05-02 16:47:18: setup.clean success --------------------------------------------------- 2019-05-02 16:47:30: case.setup starting --------------------------------------------------- 2019-05-02 16:47:31: build.clean starting --------------------------------------------------- 2019-05-02 16:47:47: build.clean success --------------------------------------------------- 2019-05-02 16:47:48: build.clean starting --------------------------------------------------- 2019-05-02 16:48:04: build.clean success --------------------------------------------------- 2019-05-02 16:48:04: case.setup success --------------------------------------------------- 2019-05-02 16:48:29: build.clean starting --------------------------------------------------- 2019-05-02 16:51:18: build.clean success --------------------------------------------------- 2019-05-03 11:36:01: case.build starting --------------------------------------------------- CESM version is cesm2.1.1-exp14 Processing externals description file : Externals.cfg Processing externals description file : Externals_CLM.cfg Processing externals description file : Externals_POP.cfg Processing externals description file : Externals_CISM.cfg Checking status of externals: clm, fates, ptclm, mosart, ww3, cime, cice, pop, cvmix, marbl, cism, source_cism, rtm, cam, ./cime clean sandbox, on cime5.6.15 ./components/cam clean sandbox, on cam1/release_tags/cam_cesm2_1_rel_26/components/cam ./components/cice clean sandbox, on cice5_cesm2_1_1_20190321 ./components/cism clean sandbox, on release-cesm2.0.04 ./components/cism/source_cism clean sandbox, on release-cism2.1.03 ./components/clm clean sandbox, on branch_tags/relcisofix.n01_release-clm5.0.20 ./components/clm/src/fates clean sandbox, on fates_s1.21.0_a7.0.0 ./components/clm/tools/PTCLM clean sandbox, on PTCLM2_180611 ./components/mosart clean sandbox, on release-cesm2.0.03 ./components/pop clean sandbox, on pop2/release_tags/pop2_cesm2_1_rel_n04 ./components/pop/externals/CVMix clean sandbox, on v0.93-beta ./components/pop/externals/MARBL clean sandbox, on cesm2.1-n00 ./components/rtm clean sandbox, on release-cesm2.0.02 ./components/ww3 clean sandbox, on ww3/release_tags/ww3_cesm2_1_rel_01 2019-05-03 11:47:00: case.build success --------------------------------------------------- 2019-05-04 09:00:22: case.run starting --------------------------------------------------- 2019-05-04 09:00:28: model execution starting --------------------------------------------------- 2019-05-04 18:37:11: model execution success --------------------------------------------------- 2019-05-04 18:37:11: case.run success --------------------------------------------------- 2019-05-04 18:37:25: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-10 08:46:49: st_archive starting --------------------------------------------------- 2019-05-10 08:47:21: st_archive success --------------------------------------------------- 2019-05-10 08:48:57: case.run starting --------------------------------------------------- 2019-05-10 08:49:03: model execution starting --------------------------------------------------- 2019-05-10 18:18:40: model execution success --------------------------------------------------- 2019-05-10 18:18:40: case.run success --------------------------------------------------- 2019-05-10 18:18:53: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-13 08:04:12: st_archive starting --------------------------------------------------- 2019-05-13 08:04:50: st_archive success --------------------------------------------------- 2019-05-13 08:05:21: case.run starting --------------------------------------------------- 2019-05-13 08:05:27: model execution starting --------------------------------------------------- 2019-05-13 17:50:27: model execution success --------------------------------------------------- 2019-05-13 17:50:27: case.run success --------------------------------------------------- 2019-05-13 17:50:40: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-14 08:01:23: st_archive starting --------------------------------------------------- 2019-05-14 08:01:56: st_archive success --------------------------------------------------- 2019-05-14 08:02:32: case.run starting --------------------------------------------------- 2019-05-14 08:02:37: model execution starting --------------------------------------------------- 2019-05-14 17:55:24: model execution success --------------------------------------------------- 2019-05-14 17:55:24: case.run success --------------------------------------------------- 2019-05-14 17:55:38: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-14 17:56:03: st_archive starting --------------------------------------------------- 2019-05-14 17:56:35: st_archive success --------------------------------------------------- 2019-05-14 19:37:44: case.run starting --------------------------------------------------- 2019-05-14 19:37:51: model execution starting --------------------------------------------------- 2019-05-15 06:02:22: model execution success --------------------------------------------------- 2019-05-15 06:02:22: case.run success --------------------------------------------------- 2019-05-15 06:02:35: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-15 06:03:01: st_archive starting --------------------------------------------------- 2019-05-15 06:03:32: st_archive success --------------------------------------------------- 2019-05-15 06:07:59: case.run starting --------------------------------------------------- 2019-05-15 06:08:05: model execution starting --------------------------------------------------- 2019-05-15 17:15:39: model execution success --------------------------------------------------- 2019-05-15 17:15:39: case.run success --------------------------------------------------- 2019-05-15 17:15:52: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-17 08:46:48: st_archive starting --------------------------------------------------- 2019-05-17 08:47:35: st_archive success --------------------------------------------------- 2019-05-17 08:48:10: case.run starting --------------------------------------------------- 2019-05-17 08:48:16: model execution starting --------------------------------------------------- 2019-05-17 08:48:37: model execution success --------------------------------------------------- 2019-05-17 08:48:37: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -p "%g:" -np 1692 omplace -tm open64 /glade/scratch/cmip6/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-SSP3-7.0Lu.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-SSP3-7.0Lu.001/run/cesm.log.5904589.chadmin1.190517-084810 --------------------------------------------------- 2019-05-17 09:00:33: case.run starting --------------------------------------------------- 2019-05-17 09:00:39: model execution starting --------------------------------------------------- 2019-05-17 18:54:49: model execution success --------------------------------------------------- 2019-05-17 18:54:49: case.run success --------------------------------------------------- 2019-05-17 18:55:02: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-18 20:12:22: st_archive starting --------------------------------------------------- 2019-05-18 20:12:57: st_archive success --------------------------------------------------- 2019-05-18 20:13:29: case.run starting --------------------------------------------------- 2019-05-18 20:13:35: model execution starting --------------------------------------------------- 2019-05-19 05:56:16: model execution success --------------------------------------------------- 2019-05-19 05:56:16: case.run success --------------------------------------------------- 2019-05-19 05:56:28: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-19 05:56:56: st_archive starting --------------------------------------------------- 2019-05-19 05:57:27: st_archive success --------------------------------------------------- 2019-05-20 08:23:56: xmlchange success ./xmlchange STOP_N=6 --------------------------------------------------- 2019-05-20 08:23:56: case.run starting --------------------------------------------------- 2019-05-20 08:24:02: model execution starting --------------------------------------------------- 2019-05-20 14:27:32: model execution success --------------------------------------------------- 2019-05-20 14:27:32: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -p "%g:" -np 1692 omplace -tm open64 /glade/scratch/cmip6/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-SSP3-7.0Lu.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-SSP3-7.0Lu.001/run/cesm.log.5982592.chadmin1.190520-082356 --------------------------------------------------- 2019-05-20 14:43:21: xmlchange success ./xmlchange STOP_N=6 --------------------------------------------------- 2019-05-20 14:43:21: case.run starting --------------------------------------------------- 2019-05-20 14:43:27: model execution starting --------------------------------------------------- 2019-05-20 20:49:55: model execution success --------------------------------------------------- 2019-05-20 20:49:55: case.run error ERROR: RUN FAIL: Command 'mpiexec_mpt -p "%g:" -np 1692 omplace -tm open64 /glade/scratch/cmip6/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-SSP3-7.0Lu.001/bld/cesm.exe >> cesm.log.$LID 2>&1 ' failed See log file for details: /glade/scratch/cmip6/b.e21.BSSP126cmip6.f09_g17.CMIP6-SSP1-2.6-SSP3-7.0Lu.001/run/cesm.log.5995346.chadmin1.190520-144321 --------------------------------------------------- 2019-05-21 08:53:15: xmlchange success ./xmlchange STOP_N=6 --------------------------------------------------- 2019-05-21 08:53:15: case.run starting --------------------------------------------------- 2019-05-21 08:53:21: model execution starting --------------------------------------------------- 2019-05-21 15:04:12: model execution success --------------------------------------------------- 2019-05-21 15:04:12: case.run success --------------------------------------------------- 2019-05-21 15:04:20: xmlchange success ./xmlchange CONTINUE_RUN=TRUE --------------------------------------------------- 2019-05-21 15:04:59: st_archive starting --------------------------------------------------- 2019-05-21 15:06:44: st_archive success ---------------------------------------------------