After 14% of job (about 400h) my task was reported completed. What happend? The log-file is:

2007-03-01 18:12:28|climateprediction.net|Sending scheduler request: To send trickle-up message
2007-03-01 18:12:28|climateprediction.net|(not requesting new work or reporting completed tasks)
2007-03-01 18:12:38|climateprediction.net|Scheduler RPC succeeded [server version 509]
2007-03-02 03:30:46|climateprediction.net|Restarting task hadcm3pbb_at4f_05778378_0 using hadcm3 version 515
2007-03-02 04:53:46|climateprediction.net|Deferring communication for 1 min 0 sec
2007-03-02 04:53:46|climateprediction.net|Reason: Unrecoverable error for result hadcm3pbb_at4f_05778378_0 (<file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_3.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_4.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_5.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_6.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_7.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_8.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_9.zip</file_name> <error_code>-161</error_code></file_xfer_error><file_xfer_error> <file_name>hadcm3pbb_at4f_05778378_0_10.zip</file_name>
2007-03-02 07:17:51|climateprediction.net|Sending scheduler request: To report completed tasks
2007-03-02 07:17:51|climateprediction.net|Reporting 1 tasks
2007-03-02 07:17:56|climateprediction.net|Scheduler RPC succeeded [server version 509]

---

From the crash log:

Model crashed: umshell1.f: ATM_DYN : NEGATIVE THETA DETECTED. GA

Model crashed: umshell1.f: ATM_DYN : NEGATIVE THETA DETECTED. GA

Model crashed: umshell1.f: ATM_DYN : NEGATIVE THETA DETECTED. GA

Model crashed: umshell1.f: ATM_DYN : NEGATIVE THETA DETECTED. GA
Fatal crash! :-(

This happens when the model decides that it\'s climate is unrealistic. There are a couple of reasons for this happening:

1) The initial parameters don\'t lead to a viable climate. One of the key goals in the project is to work out which combinations of params are viable and which are nonviable. No point restoring from backup since it\'ll always stop at exactly the same point.

or

2) A spurious floating point calculation has knocked the model off course. Happens most frequently with overclocked or overheating systems. If you have a backup it may let you continue running past the crash point. 24 hours of Prime95\'s torture test is a good way to find out if your PC is stable enough.


In both cases, the model uploads it\'s climate every model year to the project servers, so the CPU time so far isn\'t wasted.

---

I'm a volunteer and my views are my own.
News and Announcements and FAQ

According to the model page on your Account, it was:

Model crashed: umshell1.f: ATM_DYN : NEGATIVE THETA DETECTED.

This usually means an instability in your computer, causing the floating point calcs to return a negative value.
Testing your computer before running another model would be a good idea.
There are 4 README files here with hints and tips.
Crashes (Hardware section), is where you should start.

edit
Beaten to it by Mike. :)