Ode to The Code Monkey

 Yaacov Apelbaum-Code Monkey  
The Code Monkey (inspired by A Dream Within A Dream by Edgar Allan Poe)

Take another slap upon the cheek,
While slaving on this project, week by week. 
You have been wrong to work so hard,
Expecting riches and managerial regard.
Grinding out functions awake and in a dream,
Will not fetch rewards or professional esteem.

What you lack are not more lines of code, 
Rather it’s architecture and a road.
To substitute quality with speed,
Is the motto of the code monkey creed. 
You who seek salvation in RAD extreme
Will find, alas, a dream within a dream.

If you examine your latest stable build,
You will notice many bugs that haven’t been killed.
Strangely, they seem to grow in relation,
To your oversized code base inflation.
So many new features! How did they creep? 
Through scope expansion, they trickle deep.

Building good software is hard to manifest,
If you fail the requirements to first digest.
The lesson to learn from this development ditty,
Is that no matter how clever you are or witty,
If you fudge the schedule and estimation phase,
There is but one reward for you. The death march malaise!

© Copyright 2010 Yaacov Apelbaum All Rights Reserved.

Drinking on the Job and Other Vices

Yaacov Apelbaum-Drinking on the job

Working in an early stage startup can be a blast. For me, it is a most rewarding personal and professional experience. There is minimal bureaucracy to get in the way and you have the opportunity to build the product of your dreams.

True, at times a few dark clouds (like the possibility of not making payroll) may gather on the horizon, but with some ingenuity you can handle it. If you are preparing to ship version 1 of your product, the pre-launch phase or the final sprint can be an ultimate adrenaline\caffeine rush, and the chronic lack of sleep will just enhance the experience.

Working in this environment may not be for everyone. The rapid pace of change, the seemingly never ending To-Do list, and the constant improvisational nature of the work often frustrate individuals who are used to more structured development. Financial stability can be another turn-off. Most early stage startups operate on a shoe string budget using seed\angel money that has to be stretched to the limit in order to build a viable proof of concept. Once in place, the funding and development cycles are repeated in the hopes of surviving long enough to attain commercial viability. These repeating cycles, if not executed properly, can become a tiresome soap opera.

To use an equestrian metaphor to illustrate the development effort in a large organization is somewhat similar to English-style show jumping competitions. A smartly dressed rider moves elegantly between the stations (projects), always maintaining aristocratic poise and composure (an eye on features, schedule, and budget). The same event in a startup looks more like a rodeo, where the inexperienced rider jumps on the back of a wild mustang with neither saddle nor reins and spends the duration of the ride screaming, swearing, and holding on for dear life (dwindling budget and product immaturity) while the horse (the competition) simultaneously kicks, bites and tries to throw him off. Net-net, a tour of duty in a startup can resemble the life of a Caribbean buccaneer. The risks are plentiful but the rewards of a buyout or an IPO can be glorious.

In addition to operational differences, startups also tend to have a more casual and festive corporate culture. To help recruit and retain talent it is not uncommon to offer employees various soft perks like no dress code, the freedom to play interior decorator, and access to amenities such as the latest video games, quality coffee, soft drinks, snacks, and other tech toys.  Some companies, in an attempt to even further sweeten the sub-par monetary compensation will go as far as to abstain from creating work policies regarding vacation, work hours, and employee conduct.

If you are on the management team of a startup, it is tempting to convince yourself that you can make up for low base pay and a high stress work environment with theme park playfulness (a approach not dissimilar from paying for prime real estate with firewater and glass beads). But in reality, trying to convince highly intelligent people that the privilege of working in a glorified techno-utopian commune is an adequate substitute for decent wages amounts to little more than Svengali’s operatic hypnotism, it may work for a while, but sooner than later the effects wear off, leaving you with consequences like high attrition rates and chronically missed deadlines.

Firewater and Glass Beads

In one of my engagements, I had the opportunity to work with a late stage startup. The company desperately needed to mature and commercialize its product and swing to profitability. This had to be done rapidly in order to secure bridge funding.  When I first took over the technology and engineering organization, I discovered that the team had an almost 50% turnover rate and that my two predecessors had left on very disagreeable terms. Initially, I was surprised that the subject of almost every staff meeting revolved around compensation, but that matter quickly came into focus. The developers were disgruntled because, in addition to being significantly underpaid, they hadn’t received their promised bonuses (which accounted for almost 40% of their salary) yet they were still being asked to work 70 hour weeks. It was evident that all the soft perks they were getting didn’t succeed in dulling their memory of the promise.

In another engagements, the CEO refused to budge on the question of base salary and missed bonuses and argued instead that work conditions (flex hours, free sporting event passes, telecommuting, and free alcohol) were more than sufficient  compensation. On the eve of the “Mother-of-all-Sprints,” (a hellish 320 hour coding month), some team members started exhibiting disturbing behavior that included sending inflammatory emails to the entire company (see excerpt below), rowdy conduct in staff meetings, and calling in sick.


As part of the “Mother of all Sprints”, we have been asked to go above and beyond our normal dedication. We have been asked to make major sacrifices in our personal schedules. We have been asked to work harder with longer hours. We have been asked to cancel existing plans we may have put in place with our families. We have effectively been asked to put our lives on hold until the end of the sprint. And we have been asked to do this without any advance warning. Furthermore, many of us disagree with the exact product direction.

  • We disagree on the functionality necessary for the company to go forward.
  • We disagree on what the bottlenecks and limits with the current process are.
  • We disagree on the feature focus of current efforts.
  • We disagree with the release date.
  • No one asked the development team what they felt was important to accomplish.
  • No one asked the development team what features they felt were necessary.
  • No one asked the development team what they they could realistically accomplish.
  • No one asked the development team when the release should occur.
  • We have no real equity in the company. We have no guarantee we will ever receive bonus payments. We have no guarantee that bonuses will be distributed fairly, if they are ever paid out.

So I have to ask myself, what is my personal motivation for doing this? What do I get out of it?

Of course, it didn’t help that the company offered its employees a steady supply of beer and hard liquor, and even had gone as far as openly encouraging everyone to drink on the job (at the ever so popular “Tequila Fridays”). HR wouldn’t implement any conduct policies because they feared that doing so would dilute the startup experience. Not surprisingly, some employees in turn responded by drinking just a bit to much and staying home sick.

Workspace structure and policies are as mandatory as social contracts and are designed to protect us from abuse and anarchy. Hobbes observed that life under the rule of the mob is “nasty, brutish, and short.” Similarly, life in a startup modeling its governance to resemble the “Lord of the Flies” is wretched and hardly short enough.

Unfortunately, no one has the exact formula for balancing discipline with productivity. But I have found the following rules of thumb to be an affective guide:

It may be contrary to your anti-establishment philosophy, but it is in your best interest to create and maintain an orderly work place. Unfortunately, the only way to achieve this is by formalizing rules for all employees and enforcing policies like performance reviews, bonus payments, vacation and sick time without favoritism or exceptions. 

Here are my six golden rules for keeping anarchy at bay:

  1. Provide formal policies regarding company usage of licensed software. It may be true that software yearns to be free, but unless it is properly licensed it should be kept off of company computers. This may sound picayune, but using counterfeit or cracked software is a huge legal issue.
  2. Regardless of how hip and progressive it may seem, do not encourage consumption of alcoholic beverages or drugs on company premises. Every garden has its snake and this one will bite you big time. It’s only a question of time before one of your employees under the influence of something he consumed at work will become a legal liability.
  3. Do not tolerate violent or abusive behavior (such as regular use of foul language or extensive absenteeism) from any employee, including leads and management.
  4. Offer your team competitive financial compensation. If you can’t afford to pay market wages, be honest and upfront about your limitations. Don’t try to renegotiate salaries down by arguing that industry wages are inflated.
  5. Be creative about company spirit and culture. Offer your team as many soft perks as you can, but remember, perks are not a substitute for wages.
  6. Don’t exaggerate or misrepresent bonus target amounts, stock option availability, or plans for employee profit sharing. Be forthcoming about the financial state of the company and its stability. My rule is “promise only what you can deliver” and in a timely manner “deliver what you’ve promised.”

Maintaining transparency and a fair work environment are two of the most important pre-requisites for a smoothly operating startup. Once these are in place, you will discover that you are well underway towards achieving your most ambitious development goals as well.

Good luck.

© Copyright 2010 Yaacov Apelbaum All Rights Reserved.

Bells and Whistles

Bells and Whistles

Several weeks ago, my car’s dashboard informed me that I would need to “check the engine”. Naturally, I promptly brought the car to the dealership. After a quick examination (and $170 in troubleshooting fees), the dealer told me that there was nothing physically wrong with the car and that the message was due to a software bug that was easily fixable with a software patch that he would be happy to apply for as little as an additional $120.

Now, I don’t know much about the latest trends of engine design, but I am quite familiar with software bugs. Over the years I have produced several of them myself and like many other software engineers, I spend a certain amount of my waking and sleeping hours chasing them around. But contrary to my trusted dealer, I fix my bugs—to delight of my customers—at no extra charge.

Being the pedagogue that I am, I seized the opportunity to educate Randy, the senior service adviser (the title on his tag), and proceeded to deliver a short and upbeat oration on the subject of the software industry’s best practices regarding free bug fixes. My friend behind the counter seemed unimpressed and reminded me that his dealership is not in the business of selling software. He then inquired about my preferred form of payment for the pending repair.

On the way home I got to thinking about the cutthroat business model that the car industry is following. Clearly, it must be the competitive nature of the industry that forces them to squeeze profit at the margins to included even basic service like fixing their own bugs. I then dismissed any further thought on the subject. Then a few weeks ago, the news broke out that the car industry had officially joined the federal hand-out program. It was now standing on-line with a tin can in hand and singing in harmony “Brother, can your spare a dime?” with all the other financial industry hobos.

So I figured that there has to be something here that doesn’t meet the eye. Why has such a well lubricated and fabulously successful industry suddenly collapsed? Contrary to the financial sectors, their operational and risk models have not significantly changed for over 100 years. Cars today are designed and built faster than they were 40 years ago and they are cheaper to manufacture and sell.

After pondering this for a while, I realized that this failure has been long in the making and just like in many active Ponzi schemes, this one has been finally flushed out when consumer spending came to a screeching halt.

In the aftermath of World War II as manufacturing was going civilian again, the car industry more than any other sector embraced a strategy that focused entirely on building a product with the highest degree of of obsolescence (an average car would be recycled every n years) and greatest curb appeal. Traditional engineering principles like safety, longevity, efficiency and environmental friendliness were sacrificed. They were replaced with a lot of chrome, increased fender size, overpowered V8 engines and pink leather. This was the wisdom of the time; you can’t charge higher prices for less car.

In the 70’s, when the Japanese car manufacturers finally introduced a new concept of vehicles very different from the traditional tank, domestic car manufacturers after much soul searching, publicly embraced the banner of the new smaller more efficient car. In reality, however, they continued to practice the same philosophy as they do to this day. So in fact, not much has changed. Yes, the large fenders and chrome bumpers are gone, but in their place we have a whole new list of bells and whistles. Apparently, the car manufacturers have been taking lessons from the software industry and are now playing the feature functionality game.

Go to any domestic dealership and you are guaranteed to find the following list of “key” features:

  • Rain-sensing wipers
  • Heated/cooled seats and cup holders
  • V8 with extra torque
  • Extended Length
  • Remote starter

What is noticeably missing from the list is an engine and transmission that are guaranteed for 300K miles or the 80 mpg gas efficiency, both of which are easily achievable in this day and age.

Unfortunately for all of us, the car industry has spent the last 60 years mastering the fine art of consumer psychology and honing of their salesmanship skills with too little time spent on developing quality products.

Thank you very much, but I’ll have to decline! Contrary to what the focus groups are saying, I don’t need a Hummer. And no, throwing in a free electric cup warmer is not going to change my mind. I am, however, interested in a bug free safe electric car that can go 400 miles between charges if there are any takers.

© Copyright 2008 Yaacov Apelbaum All Rights Reserved.

What’s with the Chicken?

Yaacov Apelbaum-Chicken

Over the past 50 years, quality and optimization have bloomed into a mature set of practices that are now leveraged in almost every industrial and manufacturing environment. Six Sigma and TQM are just two good examples for such practices. In manufacturing, unlike software development, the relationships between project schedule, component quality, and order of assembly are clearly defined and optimized (i.e. unnecessary motion is reduced.).

For a team working on an automotive assembly line, the practice of stretching deliverables to the extent possible (so long as the assembly line continues to move) is legitimate. Unfortunately, this is not the case in software development.

As product development evolved from mechanical devices to complex software packages, some previously benign scheduling habits have become vices. One such transgression is the practices of the “Schedule Chicken”. And no, this does not refer to the cafeteria’s daily practice of serving charred poultry. The origin of the term comes from 1950’s and has been documented in the movie Rebel Without a Cause. In it, two drivers are racing their hot-rods and are supposed to jump out just before their cars go over the cliff.

The first one to jump out of the car is labeled a “chicken” while the one closest to the cliff’s edge wins bragging rights. From the psychological point of view, it is interesting to note, that the rational for the “Schedule Chicken” behavior is related to the Hawk-Dove method of conflict used by players in game theory. The software development equivalent of this occurs when two or more areas of a product team claim they can deliver their features at an unrealistically early date because each assumes the other team is stretching the predictions even more then they are.

This pretense continually moves forward past one project checkpoint to the next until just before the functionality is actually due. The more mature “Schedule Chicken” practitioner will often delay addressing the obvious for as long as possible, hoping that someone else will hit the breaks and stop their car first. The ceremony in which the team lead finally admits that he can’t deliver the features on time climaxes in a primeval ritual that is reminiscent of ancient Aztec human sacrifices, albeit it’s somewhat more painful.

In the end, it is very difficult for the failed team to recover from being behind schedule, because once the truth is out, all other teams will blame them for the delay. Even if they actually do end up beating the deadline (which is highly unlikely), it will be forever impossible for them to remove the associated stigma of having been “chickens”

So how do you combat the chicken? Here are my top indications that your project has possibly been infected with an avian flu:

  1. While discussing the production schedule people use terms like guesstimating and SWAG
  2. The proposed schedule is not based on a documented matrix of previously developed products
  3. In private conversations, cross team members challenge the validity of their own schedules
  4. The team hopes to make up for lost time by adopting silver bullet tools or technologies
  5. The amount of code necessary and it’s complexity do not match available development resources
  6. Failed smoke tests, quality of builds, and bug bounces do not match the published  status reports
  7. The team is opting to build low level functionality rather than buying it and is insisting that this will actually help accelerate their delivery
  8. Development teams are regularly burning the midnight oil, are putting an exceptional amount of overtime and appear to be worn-out
  9. Project stake holders are starting to use e-mail for the creation of a “legal trail”
  10. Team leads are showing compartmentalized understanding of the project and are unable to walk through all of its dependencies (for example, they may be insisting that it’s not necessary for them to thoroughly understand other team’s implementation details or how the entire solution is going to work)
  11. Project leadership continues to commit itself to unrealistic deliverables like “zero critical bugs” or significant performance improvements, despite having priority bugs and stability issues.

The practice of Schedule Chicken may seem harmless and in some cases may even be encouraged because it promotes survivalist behavior, but in fact, if left unchecked it can have long term devastating psychological effects on the entire team and its ability to develop efficiently.

From my experience, the best mechanisms for combating the chicken are to demand open and honest communications, insisting on peer reviews, and maintaining deep visibility into all project activities.

This is not a simple task as it may goes against the grain of the entire organization but in the end, it beats the alternatives of failure and the blame game.

© Copyright 2010 Yaacov Apelbaum All Rights Reserved.

The Überteam

The Uber Team

It’s tempting to view all development teams as homogeneous masses that operate with the same efficiency. When looking for ways to improve productivity, we tend to overlook the human factor and rush to either embrace new management methodologies or focus on incrementally improving some existing processes like requirements management or bug tracking.

Certainly, there is always room for process optimization, but ultimately, these enhancements will only translate to small gains in the team’s overall performance.

In order to realize an order of magnitude improvement, we need to focus on the ‘individual’, who in my opinion is the cornerstone of the software and hardware development process. It has been shown in numerous studies such as Steve McConnell’s Rapid Development that different teams within the same organization can have significantly varying delivery efficiencies.

So how do we explain the formation of islands of excellence within the same organization? After all, don’t all teams within the company share the same HR resources, training, tools and culture?

The short answer is that excellence in operational has a lot to do with management style, team structure, and makeup.

Building and managing strong development teams goes beyond the simple management of developers to quarterly objectives and delivering functional products. What sets apart a high performance team from the garden variety one is that in addition to delivering acceptable products, the high performance team tends to exhibit more loyalty and less splintering. It works smarter, it is more creative, more cohesive and its members are more independent. A high performance team has certain qualities and a distinct spirit which emanates from the individuals in the team.

A high performance team’s esprit de corps is derived from a strong sense of individual belonging, common culture and shared vision. Building and managing this type of a team requires a clear definition of the team’s identity, documented history of success (and failure!), the cultivation of elitism (i.e. an aggressive selection process which makes team membership a coveted privilege), stoking the competitive fire (identifying the competition and targeting it), striving to being different than others and having a reputation for taking on and delivering difficult assignments. Above all, it should be a fun and intellectually rewarding environment.

Even though high performance teams are independent of project type and technology (they are certainly not limited to early stage startups), they all share certain characteristics. I call these the Seven Golden Habits, they are:

  1. Preach and Practice Honesty and Integrity – Always insist that all team dealings and communications (internal and external) be honest and open. The primary building block of all human relationships and the cornerstone of good leadership is integrity (both professional and personal), so you must show personal integrity (in moral decisions always follow the “Et si omnes ego non” principal) and instill its importance in your directs and their team members. You can always explain incompetence and overcome it. Not so with dishonesty and poor integrity.
  2. Promote High Technical, Operational, and Personal Standards – Place strong emphasis on recruiting bright and talented individuals who are passionate about their work and are domain experts. Demand that each team member provides leadership in one or more areas. Work with each team member to develop customized training and professional growth plans and make sure that each has a clearly defined career path that goes beyond basic corporate training (plans for an advanced degree, leadership opportunities, professional certifications, etc.). Inspire them to continually learn so as to further excel. Organize training in emerging technologies, standards and best practices. Promote industry visibility for team members by encouraging them to present papers at conferences and to publish. Public visibility for individual members = visibility for you and the entire team.
  3. Emphasize Accountability and Responsibility – Carefully delegate responsibility and provide your directs and team members with ample opportunities for exposure and high visibility success. Never micro manage! At the same time, expect all assignments to be completed promptly and with a high level of quality. Do not tolerate procrastination and repeated failure to deliver projects on time, over budget or with inferior quality. Provide talented team members with frequent leadership opportunities. Generously give credit where it is due and swiftly sanction habitual underperformers.
  4. Promote Cultural Diversity, Sociability and Friendliness – Recruit heterogeneous individuals (promote cultural, racial and gender diversity). Consider the stimulating exchange of ideas between team members critical for team success as well as for your own personal growth as a manager. As such, encourage the establishment of forums and strategies to build strong collaborative teamwork, some of which may include: inner team mentoring and tutoring, brown bag technical luncheons and sponsorship of after work activities. Do not tolerate any abusive behavior within your team. Get to know each team member personally. Learn about their hobbies, their family and their personal concerns.
  5. Stop the Buck and Delegate – Serve as a single executive focal-point for your management and directs (handling both inbound and outbound escalations). Build, motivate and promote inner team leadership and a successful culture that is based on mutual benefits. Empower direct reports to contribute to the development strategy, while insuring that their action plans and implementations are based on the official strategic goals and well defined market opportunities.
  6. Develop a Competitive and Rewarding Workplace – Institute spot cash bonus programs and prizes (gift cards are a favorite), permit a certain amount of telecommuting when possible, subsidize technical books, on-line education, and hardware and software for personal use, place strong emphasis on social bonding (including after work hours activities and membership in online social networks).
  7. Deal with Ambiguity Effectively – Due to their intangible nature, ambiguity and uncertainty can have devastating effects on any development team. Ambiguity tends to increase with project size and complexity and can stem from many sources that may not be under your team’s control (i.e. budget, schedule, requirements, technology, strategy, etc,). You should expect ambiguity as a given and develop approaches to elevate and counter its effect on your team’s ability to shift gears quickly without having all the facts.

Some of the coping techniques I have used to deal with ambiguity include:

  • Avoid “analysis paralysis” and “deadlocks” – Even though most of us subscribe to a structured, orderly and predictable management methodology, in the absence of more information, you should feel comfortable relying on your team’s advice as well as your personal experience and intuition to reach a decision without endlessly pondering all of the options.
  • Keep clear from the tried-and-true trap – Under conditions where the work threatened by various conditions of uncertainty, traditional solutions may not be feasible, you should push the team to innovate, think out the box, and take calculated risks to insure that business objectives are met.
  • Manage the effects of ambiguity on the team – Understanding the relationship between anxiety, impatience and uncertainty, you should work to develop an effective communication plan and control the effects of uncertainty on the team by: providing daily communication (meetings, e-mails, briefings, etc.), fighting rumors, requiring every team member to provide constructive suggestions, eliminating team splintering and factioning, speaking externally with a single voice, and avoiding finger pointing and blame.

These rules work well whether you are first forming a new team or transitioning forward a mature (and perhaps stubborn) one. Whatever the constellation of your corporate environment and culture is, these are tried and true methods that will yield significant improvements in operational efficiency and product quality.

Contrary to the old adage, you certainly can (and should!) teach an old pooch new tricks.

Good luck!

© Copyright 2008 Yaacov Apelbaum All Rights Reserved.