Friday, May 25, 2007

Memoirs of my time at The Company

I finally got fed up with The Company and told them to frak-off. My last day will be May 31st and I'll start my new job some time in June.

Things have been going downhill at The Company for quite some time. Actually, I don't think they were ever going uphill. When I left Lunar Logic, I thought working for a big company would be great, that they'd have everything figured out, decent processes in place, and that I'd find a bigger pool of brains to pick and find more mentors in a larger community. Turns out that I was wrong. Bigger doesn't mean better, and I was not very impressed with the quality of the people. For awhile I liked the money, but I've come to learn that making lots of money doesn't make up for all the stress I've endured or the personal time I've been missing.

My first three months at The Company were spent in crisis mode. I was shuffled from conference room to conference room and put on projects that were already behind schedule and needed help. AccountLogin was Jen's creation and involved a major rewrite of the entire front-end controller along with changes to the main path through the site which was predicted to increase order completions and revenue. We worked long hours in close quarters and then proudly rolled it out to production. Part of the project required that the Business have built in switches to control how many customers used the new path versus the old path and that they would be able to closely monitor customer behavior and make changes to the traffic split as necessary. Within a week of going live the Business determined that their predictions had been wrong since less order completions were happening on the new path. They panicked and turned off the new path completely, in effect moth-balling AccountLogin. The project was viewed as a failure despite the effort it took to get there.

Next I was put on Cashbox-BP. This was an ambitious project which was already falling apart due to lack of competent leadership. The previous dev lead had left on extended medical leave of absence three weeks into the project. The backup lead was in over his head. I came in and took over a large chunk of the complicated work that no one else wanted to deal with. I didn't know any better at the time. Basically the project involved moving customer billing and account information over to a remote system called EBE. It also added a new feature of allowing customers to auto-renew their subscriptions every year unless they opted-out. The Business expected to make millions off of that one change alone. I ended up re-coding most of the services that communicated with EBE and debugging all those issues. Working with outside consultants on this project proved to be a painful experience. Cashbox-BP finally rolled out and The Business threw us a little party because they started seeing the numbers go up as expected.

Once those projects were done I was only able to take a short breather and get barely acquainted with my office which had been sitting empty during that time before the whole development team was chased out of their offices and herded into acres of identical gray cubicles.

My next project was a follow-up to the last one, Cashbox-Maintenance. This was were we cleaned up all the things that weren't quite right the first time around, and implemented a few new features that got left off the last release. This time around we had a team that was experienced with the last round of the project, but we still had to coordinate with developers from EBE and Vindicia, which was as painful as ever. Finger pointing was rampant whenever something went wrong and usually it turned out the be Vindicia's fault, but you had to prove it to them first. Once again I was given the largest chunk of work and the most complicated tasks, the price I paid for proving myself competent. Since I made the most changes I began running code review sessions as well. As the holidays rolled around I started to act more like a lead than the actual lead since he was on vacation most of the time. By the time Cashbox-Maintenance went live I was effectively the lead on the project. My manager was grooming me for being a lead on the next project as well because he we getting positive feedback from the project managers about me. He promised me a promotion at my next review, in six months, if I kept doing the extra work.

Before we got a chance to release Cashbox-Maintenance the live site started having problems. A major bug was missed in the last release and starting to show up on production. We had two weeks of frantic emergency releases to fix the issue and push out another high priority feature. Cashbox-maintenace got delayed but eventually went live.

Almost immediately I was made lead of Short-term AccountLogin Forgot Password (STAL-FP), a follow-up project to the "failed" AccountLogin project. The idea was to fix some of the easy things that were determined to be wrong with the current storefront site which could positively effect revenue. Research showed that customers who came to the site and had forgotten their password got frustrated with our stupid process for resetting it and eventually dropped off completely. Jen designed a solution, I led the development, and India did most of the work. Everything seemed to be going smoothly, although the project scope got larger the longer the development went on and we had to include more and more unrelated maintenance tickets in our release. The business had recently decided that we needed to be able to release maintenance tickets every two weeks. That made it difficult when the development time was scheduled over six weeks to test my changes in isolation.

Halfway through the successful development of STAL-FP I was handed another high priority project to lead, PacificPlatform. Leading them both at the same time proved to be very difficult and both projects suffered from my lack of attention. When I complained about the workload I was reassured that if I kept up the good work I'd have a promotion waiting for me at my review date, which was still five months away. PacificPlatform was another ambitious project, a complete redesign and rewrite of the catalog system which would support two reseller store platforms at once (code named Atlantic & Pacific). Eventually the project got renamed to GlobalCommerce. As that project was ramping up STAL-FP was getting ready to be released. Unfortunately we had to merge with two other projects at the last minute, one of which was a complete rewrite of service metrics. We crossed our fingers and hit the deploy button.

When STAL-FP hit production suddenly servers started to crash. Everyone panicked. All other work stopped as the entire development team tried to figure out what went wrong. The decision was made to rollback, which doesn't happen very often. Hundreds of hours were spent researching the server crashes and trying to reproduce the issue in development. Eventually we found it, but not before we had royally screwed up the code base by trying to rollback merges and remove service metrics while cross-merging STAL-FP code into trunk. Basically it got all frakked up. We spent extra hours trying to untangle the mess that was made. Everyone suffered. STAL-FP was considered a failure.

Once STAL-FP was abandoned I was at least able to focus on GlobalCommerce, which had already gotten started without me. I did my best to catch up and implement weekly code reviews to keep everyone on task. Things seemed to be going well for awhile as I got a handle on everything. I wasn't too happy with the team of developers I was given, but I did my best to delegate the work in such a way that the weakest members had the least impact on the project. One of them was the lead that had just returned from his extended medical absence and there were some doubts about his ability to contribute. I gave him what I thought would be the simplest and most isolated tasks. As the work progressed it became apparent that our biggest challenges would be changes necessary to the configuration of the various environments our new code would be running on. I had to rely heavily on others to advise on the necessary changes since I'd had no experience with that portion of the application.

Code reviews went well and all developer testing results showed we were ready for QA, but QA wasn't ready since they were all busy testing the three releases going out before us. QA found no bugs in our project so we hit code freeze and entered the release-to-production pipeline where it is much more problematic to fix bugs and push out new builds. I spent a grueling week in external code review and then immediately left on a much needed week vacation with Kat to New Orleans.


Once I got back we were in RTP and QA started finding bugs. Unsurprisingly the majority of them were configuration and environmental issues, but a good number of them were code issues in the areas I had given to my weakest developers. We spent twice as long as we should have in RTP, fixing an endless sequence of errors. Then load test results were coming back as showing a sudden drop in performance. Countless hours were lost investigating the performance issues and rerunning load tests late at night. It turned out to be an issue with EBE (big surprise), which I had suggested at the very beginning, but my theory had been discounted since EBE said they hadn't made any changes on their side. By this point almost everyone involved in this project was not happy about it, especially me.

Oddly enough, during the worst part of the GlobalCommerce project two new job opportunities opened up to me. Language Learning Solutions had courted me a year ago when I was planning to leave Lunar Logic. They contacted me just before I left for New Orleans and we set up a meeting for when I got back. At the same time some guys from Lunar Logic had started a new business, Emberex, and really wanted to bring me on. I had lunch with them as well and we talked about the possibilities. I was dubious at first because it seemed like a risk, but I knew I wanted to work with my friends again and escape the insanity that was going on at The Company. As I became more exhausted with work, the idea of changing jobs got more appealing. I started to update my LinkedIn profile and solicited recommendations from current and former co-workers. To my pleasant surprise I received a flood of glowing recommendations. That really bolstered my confidence at a time when I really needed it.

Around this time I learned my year review and potential promotion was going to be delayed an additional three months. The only compensation I had received for all my extra effort during the last year was an A++ award that came with a $1000 bonus, which was about a 1% bump in my pay for the year. The promises of raises and promotions were losing their power over me, and the money ceased to be the main deciding factor in my choice of jobs. LLS offered a higher salary, but wanted me to come in as an executive and build their IT department. I felt like I would be just as stressed in that position, if not more, than I already was at The Company. The guys at Emberex offered a reasonable hourly wage, which would be a welcome respite from the abuse of the salary system I was currently experiencing, and a narrowed set of roles and responsibilities. Plus they had some things to offer that LLS and The Company couldn't, a sweet downtown location, a small group of competent co-workers that I already knew and liked working with, and the opportunity of working on diverse projects and new technologies. It became clear that I wanted to join them, even if it meant losing some of the valuable benefits that I was used to at The Company.

I could see myself missing some of my co-workers that I'd grown to like and respect: Charles, Peter, Tara, Michael, Jen (of course), Steve, Imran, Martin, Ross, and David. I'd learned a lot from Charles and Steve at the foosball table. I'd miss the cheap and fast food at the cafeteria, I was getting very used to hot breakfast sandwiches and fresh Starbucks coffee every morning, but that was contributing to my increasing waist size and the general degradation of my health. I wouldn't miss much else.

I let my manager know I was considering leaving, then when he couldn't convince me to stay I called Emberex and accepted their offer. I sent my resignation letter to my manager, then I turned down the offer LLS had given me as graciously as I could. I knew I would probably be working with them at some capacity through Emberex very soon. Suddenly things seemed to be moving very quickly. Everyone was shocked to hear I was leaving in two weeks and I had people stopping by, emailing and instant messaging me wondering why I was leaving and where I was going. I tried to be as diplomatic as possible and talk about how I couldn't pass up the opportunity to join a start-up with my friends rather than focus on how frustrated I was with the work and culture at The Company. I assured everyone on the GlobalCommerce project that I was going to make sure we finished the project in a good way before leaving and that my leaving was nothing personal, which wasn't entirely true since some of the people on the project contributed greatly to my desire to leave.

Immediately after putting in my notice I started to feel better. My anxiety about making a decision about my career melted away. My stress level over work related issues lessened since I knew I wouldn't have to deal with it for much longer. I stopped taking my laptop home and stopped caring about emails sent after-hours or missing lately scheduled meetings. It's amazing what not caring can accomplish.

So now I've only got 3 days of work left and I'm feeling really good about the future. I get to take about two weeks off from work between jobs. Hopefully I can get some things done around the house that I've been meaning to, like painting. That was my plan the last time I changed jobs at least. I've got a nice long 3 day weekend and I might get a chance to play my Game of Thrones Kat got me for my birthday. Maybe I'll even do some yard work. Whatever I do I'll be happier than I've been in the last six months because I made decision that has my health and well being my priority.