Mark (poisonkitty) wrote,

  • Mood:

Hell: A Tiered Approach

Lately, I've been peeling back the onion layers of just how stressful a job can get. I'm nowhere near the worst it's ever been (or could be), but I'm definitely spending too much time working.

The most stressful project so far had to be 50 cent flights dot com, where we had daily "sales" of 50 flight tickets for 50 cents each. Every day before, during and immediately after, there were different kinds of stress. "Is everything going to work today?" "Will hackers game the system again and win all the flights?" "Will today's fix for yesterday's hack attempt break anything?" "OMG, will our servers crack under the load today?" "Did we actually collect all of the data today or is some missing?" "Did we accidentally sell more tickets than we have to offer?" I had a row of monitors with monitoring tools running for the duration of the sales (10-30 mins, usually). Project managers would be pacing back and forth between cubes where everyone was monitoring different servers. We usually had 50-100k page views per event spread across 15 minutes or so.

That was the first time I ever saw a webserver with a load average over 70. That's not 70% busy, that's an average of 70 processes waiting for CPU time. I would smoke a cigarette before each day's flight sale (depending on the time of the sale that day). Afterwards, some days, the team would have some medicinal booze if it were particularly stressful. If we hadn't built that website in a few weeks, I don't think it would have been so stressful. We were working out bugs in the system weeks after it was live in production. In the 2 months that this project was online, we only had 1 meltdown - on the first day before we knew what kind of traffic to expect. That project has tempered my stress handling abilities, I think.

The current project is similarly insane. They want "Web 2.0 YouTube and TypePad with a calendar, user-submitted videos, a custom video player, "viral" stuff, comments, stories, blogging, rating, etc., etc..." and we were given 3 weeks to build it in. It is supposed to go live on the 19th and I'm still working on it. This is my second weekend in a row to be working on it. I think I must take tomorrow off, even if it means that the project is late. The amount of time since my last weekend is inversely proportional to the quality of my work. If I were to work tomorrow, I would probably start coding in reverse at some point. My only solace is that once the first release comes out, I can start telling people to fuck off and wait while I take my time with the rest of it.

After laying out all of the requirements, we split it up into several releases with all but the first having floating delivery dates. Unfortunately, 80% of the features got crammed into the first release. This first release represents 3 weeks of a 3 month project. Hence, my expectation that the remaining 20% of the features will be a vegan cake walk in comparison. I'm glad we have people selling these stupidly trendy "Web 2.0" projects, because it's interesting to do things like server-side video conversions and flash player integration. I just don't really see the business value in most of it.

To top off the stress factors, they're sending out 200,000 emails to users to submit videos and such. There will be TV commercials driving traffic to this site, and they are strongly considering a superbowl commercial. Superbowl commercials are really in their own league as far as bandwidth considerations. That beating won't happen till next year, though, thankfully.
  • Post a new comment


    default userpic

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.