Pride in Teamwork
Last night my team deployed the fall release of our software to production. Engineering had been working on this release for the past 3-4 months, and the actual deployment was the culmination of that effort.
For those that may not know what's involved, here's what typically happens during release night. There's a team of us here all evening: the entire QA team, a release engineer, a DBA, and all the development leads. Typically, there's a lot of prepwork and planning that happens the day before and the day of the release, and a whole bunch of activity the night of the deployment.
The actual software deployment starts at 8pm and typically takes about 30-45 minutes if there are no problems. For the most part, most of the team waits as the release engineer and DBA deploy the actual code. After the deployment finishes, all the software is turned "on", and QA goes to work. They run automated regression tests, verify the results, and run the manual tests for new features. If all goes well, QA is done in about an hour, and everyone can go home.
However, Murphy's Law always seems to take over, and something typically goes astray. That's why all the development leads are here -- to investigate and resolve any problems that come up. This can take anywhere from a half hour to several hours to resolve all the issues.
We had decided that due to client impacts, releases had to happen on Friday evening. I don't know about the rest of the team, but I'm guessing like me the last thing anyone wants to do on a Friday night is hang out at work. If you did the math from the times I gave, this means that we don't typically get out till 1-2am.
And 2am is the time we completely finished last night. We ran into a couple of glitches -- one related to a SQL Server bug, and another on our deployment program (which we had not run into in any other environment). Last night, I was admittedly annoyed that we had these problems. But today, after having had some sleep, coffee this morning, and time to think about what happened, I think we handled ourselves very well.
As I started thinking about it further, I reflected back to all 18 releases and realized we've done pretty well.
For those that may not know what's involved, here's what typically happens during release night. There's a team of us here all evening: the entire QA team, a release engineer, a DBA, and all the development leads. Typically, there's a lot of prepwork and planning that happens the day before and the day of the release, and a whole bunch of activity the night of the deployment.
The actual software deployment starts at 8pm and typically takes about 30-45 minutes if there are no problems. For the most part, most of the team waits as the release engineer and DBA deploy the actual code. After the deployment finishes, all the software is turned "on", and QA goes to work. They run automated regression tests, verify the results, and run the manual tests for new features. If all goes well, QA is done in about an hour, and everyone can go home.
However, Murphy's Law always seems to take over, and something typically goes astray. That's why all the development leads are here -- to investigate and resolve any problems that come up. This can take anywhere from a half hour to several hours to resolve all the issues.
We had decided that due to client impacts, releases had to happen on Friday evening. I don't know about the rest of the team, but I'm guessing like me the last thing anyone wants to do on a Friday night is hang out at work. If you did the math from the times I gave, this means that we don't typically get out till 1-2am.
And 2am is the time we completely finished last night. We ran into a couple of glitches -- one related to a SQL Server bug, and another on our deployment program (which we had not run into in any other environment). Last night, I was admittedly annoyed that we had these problems. But today, after having had some sleep, coffee this morning, and time to think about what happened, I think we handled ourselves very well.
As I started thinking about it further, I reflected back to all 18 releases and realized we've done pretty well.
- We are 18 for 18 in meeting our deadlines. For anyone that has ever worked in software engineering projects, this is a phenomenal statistic.
- Though we plan rollbacks in case anything disastrous happens during a software release, we have *never* rolled back, i.e. our software releases have never failed.
- The engineering team is extremely dedicated. Everyone stays till the job is complete. And I don't hear any whining. If there are issues, the most important are fixed before we go home.
- The team has a really good sense of humor. Last night, as we were trying to resolve problems, everyone was still working hard, but still cracking jokes.
- We keep getting better at releases. Sure, there were problems last night, but they were problems that we had never seen before. We learn from each one and continue to improve.
Congrats! Glad to hear that things went well.
Reply to this