Posts by Gene

1) Questions and Answers : Wish list : Smaller Work Units (Message 46203) Posted 13 May 2013 by Gene Post: It is viable to have smaller work units without comprising the integrity of the data. Since the model has to run for a continuous period of time, it cannot be parallelized among many computers simultaneously, but it can be swapped between one computer and another at arbitrarily short prescribed checkpoints as long as all the associated data is passed along. So, for example, with a 40 year run (2020-2060), Ivan can crunch year 2037, after the server gets the data from Bob, who crunched 2036, etc. After all the data is received from those 40 chronological work units it is stitched together and is entered as part of the model ensemble. This has the added benefit of being able to set up short deadlines for the work units, and sending out the work unit again if it isn't crunched in time. This might also save a lot of time if a model spins out of control; rather than running for another 400 hours, the server performs a quick reality check of the parameters before sending out further sequential work units, and if they exceed certain thresholds informs the modelers. Setting up the modeling in this way would increase the server traffic by the factor proportional to the decrease in work unit time (maybe a factor of 100 would be ideal), which might put a strain on the server hardware. This reduction in model length could be very easily implemented. It seems it would go a long way to solving user frustrations with running a model for 800 hours only to have it end in a computation error, or upload error, etc. (I have had a few of those since I have been contributing my computing time for the last 5 years.) I'm a climate scientist at the Lamont-Doherty Earth Observatory, so I don't work with these models directly, but I also understand very well how breaking a model into subcomponents can be done easily. If anyone wants to discuss with me how this is impossible, I'd be glad to chat about how to do it (pm or email me at Lhenry@ldeo.columbia.edu) . ~Cheers