climateprediction.net home page
Task 16033613

Task 16033613

Name hadcm3n_o0ie_2140_40_008269678_3
Workunit 8424802
Created 24 Sep 2013, 22:21:36 UTC
Sent 24 Sep 2013, 22:21:57 UTC
Report deadline 25 Dec 2013, 5:49:08 UTC
Received 15 Oct 2013, 0:50:10 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1292569
Run time 15 days 5 hours 47 min 42 sec
CPU time 14 days 17 hours 57 min 41 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 2.60 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>7.0.27</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
12:09:08 (20658): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
05:12:01 (16164): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
05:14:19 (16749): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 63 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 65 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
05:17:00 (16761): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:19:57 (16781): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:22:30 (16795): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:24:42 (16809): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:27:39 (16821): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:29:46 (16835): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:32:13 (16849): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:35:06 (16862): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:37:43 (16875): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:39:50 (16888): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:42:26 (16902): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:44:44 (16915): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:47:06 (16927): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:49:38 (16942): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:51:55 (16955): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:53:57 (16969): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:56:49 (16981): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
05:59:41 (16994): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:02:04 (17008): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:05:00 (17022): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:07:08 (17035): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:09:59 (17048): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:12:57 (17062): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:15:14 (17075): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:17:50 (17089): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:20:12 (17107): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:22:20 (17121): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:24:52 (17135): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:27:09 (17151): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:29:37 (17165): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:32:28 (17178): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:34:31 (17192): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:36:58 (17205): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:39:20 (17218): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:41:36 (17232): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:43:58 (17245): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:46:26 (17257): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:49:08 (17272): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:51:26 (17285): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:53:33 (17299): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:56:24 (17311): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
06:59:16 (17324): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:01:38 (17338): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:03:50 (17352): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:06:17 (17365): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:08:54 (17378): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
07:11:12 (17392): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	08:52:27 PM	No files match the supplied pattern.
MainError:	08:52:27 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
MainError:	11:02:11 AM	No files match the supplied pattern.
MainError:	11:02:11 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
MainError:	10:36:32 PM	No files match the supplied pattern.
MainError:	10:36:32 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	01:03:33 PM	No files match the supplied pattern.
MainError:	01:03:33 PM	No files match the supplied pattern.
MainError:	01:26:25 AM	No files match the supplied pattern.
MainError:	01:26:25 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	01:46:30 PM	No files match the supplied pattern.
MainError:	01:46:30 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	03:04:40 AM	No files match the supplied pattern.
MainError:	03:04:40 AM	No files match the supplied pattern.
MainError:	02:22:56 PM	No files match the supplied pattern.
MainError:	02:22:56 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	06:39:39 PM	No files match the supplied pattern.
MainError:	06:39:39 PM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
MainError:	08:41:45 AM	No files match the supplied pattern.
MainError:	08:41:45 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Error: Input file: dataout/o0ieko.pjz2c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieko.pjz2c10
Error: Input file: dataout/o0ieko.pjy1c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieko.pjy1c10
Error: Input file: dataout/o0ieko.piz2c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieko.piz2c10
Error: Input file: dataout/o0ieko.piy1c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieko.piy1c10
Error: Input file: dataout/o0ieko.pfz2c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieko.pfz2c10
Error: Input file: dataout/o0ieko.pfy1c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieko.pfy1c10
Error: Input file: dataout/o0ieka.phz2c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieka.phz2c10
Error: Input file: dataout/o0ieka.phy1c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieka.phy1c10
Error converting file to netcdf: dataout/o0ieka.ph11c10
Error: Input file: dataout/o0ieka.pgz2c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieka.pgz2c10
Error: Input file: dataout/o0ieka.pgy1c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieka.pgy1c10
Error converting file to netcdf: dataout/o0ieka.pg11c10
Error: Input file: dataout/o0ieka.pez2c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieka.pez2c10
Error: Input file: dataout/o0ieka.pey1c10 is not a valid UM file.
Error converting file to netcdf: dataout/o0ieka.pey1c10
Error converting file to netcdf: dataout/o0ieka.pe11c10
MainError:	01:10:17 AM	No files match the supplied pattern.
MainError:	01:10:17 AM	No files match the supplied pattern.
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Numerical result out of range
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

Model crashed: STWORK  : I/O error - PP fixed length header                                                                                                                                                                                                                    tmp/pipe_dummy                                                                  2048    

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Numerical result out of range
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

Model crashed: STWORK  : I/O error - PP fixed length header                                                                                                                                                                                                                    tmp/pipe_dummy                                                                  2048    

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Numerical result out of range
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

Model crashed: STWORK  : I/O error - PP fixed length header                                                                                                                                                                                                                    tmp/pipe_dummy                                                                  2048    

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Numerical result out of range
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

Model crashed: STWORK  : I/O error - PP fixed length header                                                                                                                                                                                                                    tmp/pipe_dummy                                                                  2048    

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Numerical result out of range
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

Model crashed: STWORK  : I/O error - PP fixed length header                                                                                                                                                                                                                    tmp/pipe_dummy                                                                  2048    

BUFFIN: Read Failed: Inappropriate ioctl for device
BUFFIN: C I/O Error feof - Unit 64 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 66 - Return code = 1

BUFFIN: Read Failed: Invalid argument
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

BUFFIN: Read Failed: Numerical result out of range
BUFFIN: C I/O Error feof - Unit 67 - Return code = 1

Model crashed: STWORK  : I/O error - PP fixed length header                                                                                                                                                                                                                    tmp/pipe_dummy                                                                  2048    
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Oct 2013 01:33:38 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 777,600 1,271,534 1.6352
13 Oct 2013 08:59:21 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 751,680 1,230,839 1.6375
12 Oct 2013 21:19:25 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 725,760 1,190,257 1.6400
11 Oct 2013 14:29:50 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 699,840 1,149,715 1.6428
11 Oct 2013 03:07:14 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 673,920 1,109,448 1.6463
09 Oct 2013 13:51:16 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 648,000 1,068,869 1.6495
09 Oct 2013 01:29:22 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 622,080 1,028,733 1.6537
08 Oct 2013 13:04:48 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 596,160 987,865 1.6570
07 Oct 2013 22:37:43 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 570,240 947,136 1.6609
07 Oct 2013 11:22:21 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 544,320 906,641 1.6656
06 Oct 2013 20:52:09 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 518,400 865,994 1.6705
06 Oct 2013 07:08:58 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 492,480 825,222 1.6756
05 Oct 2013 17:51:55 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 466,560 784,436 1.6813
05 Oct 2013 04:07:06 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 440,640 743,713 1.6878
04 Oct 2013 15:12:18 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 414,720 703,145 1.6955
04 Oct 2013 03:43:39 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 388,800 662,850 1.7049
03 Oct 2013 04:59:25 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 362,880 621,699 1.7132
02 Oct 2013 17:12:42 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 336,960 580,139 1.7217
02 Oct 2013 06:17:56 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 311,040 541,104 1.7397
01 Oct 2013 14:33:35 1292569 16033613 hadcm3n_o0ie_2140_40_008269678_3 285,120 497,281 1.7441


©2024 climateprediction.net