I am just posting this to vent. . .which I think we all need sometimes!
I am working on a major data warehouse. I am essentially the sole coder with generalized support from the programming department. This has been the biggest sql based project I have ever worked on - and the first data warehouse. Man have I learned alot!
I have been struggling with load times - the balance of resources, data update, and indexes. We are talking a good 10,0000+ fact records a day. Long story short, I spent all day yesterday reworking the timing of the index builds, setting up a rebuilt script, and testing load times. Since some of the scripts take over an hour to run this is a tedious part of the process. I can’t completely leave the computer, but I have to stay around in-case there are any errors. I also have to decide when its taking too long. I finally got things processed in a time frame that I think is acceptable. And I did so in time for the nightly load to kick off - finishing up around 11:30pm.
This morning I excitedly checked in.
Crap - I had forgotten to remove one of the store procedures. So the second step in the job (the step I had been testing the previous day) did not complete. Crap crap crap.
Anyway. I corrected the error and I am running the step again. Fingers crossed it should finish in approximately 7 hours. Then I can actually do the work that is billable for this customer - rewriting their original report scripts to point to the new data warehouse.
Anyone else have a vent? Feel free to follow up! You are not alone!

