Uncovering the Dark Secrets of Dubious Software Startups

Yaacov Apelbaum-Lavitation API and Bridge for Sale

Maybe you are thinking about buying a new technology platform or investing in a software startup. Following industry practices, you will likely conduct some form of due diligence before you make your big move. This may include interviewing members of the management, technology and finance teams. You may also conduct operational audits, review sales figures, talk to customers, and check for references.

All advisable but in the end, you will still be left with a certain amount of nagging doubt. After all, how do you really know what this company’s true technology abilities are? How can you tell with a high degree of certainty that you are not buying the Brooklyn Bridge equivalent of some useless/over-hyped software? In today’s frenzied Internet of Things, mobile and Big Data buzz-ridden world, sometimes it seems as if the sky is the limit. To the uninitiated, it is exceedingly difficult to tell the difference between a solid early stage software idea and a useless concept professing to be the next big, anti-gravity SaaS solution.

I know. You are probably asking yourself: how difficult can it be? After all there are numerous simplified due diligence guides that answers questions like:

●  Does the company really own its supposed product?
●  Is the technology integrated/constructed in the right way?
●  Can their technology scale?

Unfortunately, when you are evaluating a technology potential, you may find that the answers to such questions are fuzzy and not always easily discernable . So before you make your investment decision based on some generic checklist, you may want to consider the following tale about the rise and fall of a flying super hero in tights.

In 2010, following the meteoric success of the Spider-Man movie franchise—which grossed over $2.5 billion worldwide—a stage adaptation entitled “Spider-Man: Turn Off the Dark” arrived to Broadway. The investors spared neither expenses nor talent in pouring over $75 million into the production in hopes of recreating the movie magic and revenue.

To stay true to Spider-Man’s legacy, the play executed some complex aerobatics sequences and flight scenes across the stage.  These stunts quickly gained notoriety as the show became plagued by accidents.

Some of the more noteworthy injuries included:

– Stunt double Kevin Aubin broke both wrists when he was catapulted from one end
of the stage to the other

Brandon Rubendall broke a toe that same month doing the same stunt as Aubin

Natalie Mendoza, who played villain Arachne, suffered a concussion when she was struck in the head with a piece of equipment

Carpio, Mendoza’s replacement, suffered a neck injury after a battle scene with

– Stuntman Christopher Tierney fell 30 feet into the orchestra pit suffering a fractured skull, a fractured shoulder blade, four broken ribs, and three broken vertebrae

Daniel Curry, a stunt double, got his right foot stuck in a stage lift and then a trapdoor closed on the foot, breaking the foot and both of his legs, necessitating amputations

Yaacov Apelbaum - Spiderman fallThis reads more like an account from the trenches of Verdun than a Broadway musical. Despite the carnage, the performances went on with regular venue changes and constant retooling of the storyline and musical score.

Even negative press reviews such as the “Pigs Will Fly Before Spider-Man Recoups $65 Million Costs” could not stop the show.

Finally last month, the producers announced that they plan to end the production in January 2014, the main reasons being falling ticket sales and—not surprisingly—the inability to get injury insurance for the cast.

In the end, the show will have run for over three years and will have lost an estimated $60 million.

So, what went wrong? Why did life fail to imitate art? It seems that on the live stage, the same stunts that were so easy to achieve in virtual CGI failed miserably when ported to the physical world. Why wasn’t it obvious from the start that the Spider-Man storyline could only work in the pages of comics and on the silver screen?

The investors behind the Broadway adaptation were seasoned entertainment entrepreneurs. Before committing funds to the project, they conducted their due diligence and found the venture to be worthy. Yet over a period of 3 years and despite watching repeating cycles of misfortune, they failed to pull the plug. Apparently, hope springs eternal—at least in the investor’s breast. Sometimes, even though red flags may be staring you right in the face, you can still miss all of the warning signs.

Yaacov Apelbaum- Spiderman flyingGlen southern - Fat Spiderman

Image 1: Spider-Man Planned vs. Actual

Over the years, I have  conducted due diligence on various software partnerships, acquisitions, and investment opportunities. It turns out that questions like: ‘how scalable/portable is this solution?’, or ‘how valuable is the code?’ are not only difficult to answer but often irrelevant. Yaacov Apelbaum - Dehydrated Water

And just like in the example of the Spider-Man fiasco, even seasoned professionals can fall victim to a well rehearsed pitch presented by a charismatic team of snake oil salesman who can sell you dehydrated water without even blinking.

In many ways, evaluating an investment opportunity in software is like a game of cat and mouse. Your evaluation will involve constant pursuit, near captures, and repeated escapes. You will have to sift through piles of partial facts, exaggerations, and in some cases even deliberate misinformation.

This is to be expected.  No cause for alarm though. Here is a three phase approach to conducting due diligence effective enough to help strip the thin veneer of pretense so that you can get deeper insight into how your potential acquisition functions and what its possible soft spots are.

Before you start probing any soft spots, though, you will need to get the regular DD action items out of the way.  Conduct some background research and get Intel on the  following:

●  Litigation (are the company and/or it’s principals in court for any reason?)
●  Costs to operate the business for the next 12 months based on current burn down rate
●  3rd party licenses and vendor agreements (both, in terms of income and expense)
●  Customer base, future growth projections, and teaming agreements
●  Forecasted capital investments (what are the costs of boarding one new customer?)

Now that you have the basics you can proceed to look for chinks in the armor.  Schedule some face time with the technology team, including: architects, operations, IT, development,  QA, etc.  It is important that you conduct both group and personal interviews with these individuals because the group dynamics will effect the detail and quality of the answers you get.

The topics that I find to be the most illuminating include:

Management Pedigree – Find out if the the leadership team has prior successful entrepreneurial experience. Take the time to check them out on-line before meeting them face to face. (LinkedIn is a great source for this.) Each technical leader should have at least five to seven years of “specific and proven” experience in the areas that the company is trying to innovate (i.e. cyber security, analytics, etc.). Having general practitioner without deep domain experience will dramatically decrease the chances of their success because they will have to learn on the job and this will undoubtedly be time consuming and error prone.

Also, look into the tenure of the key members on the technical team. Has the CTO or VP of engineering been with the company from the get go?  Is there rapid turnover in any of these key positions? A revolving door syndrome could be an indication that the company failed to mature their technology and is trying to bridge the gap by searching for “the one” who will save them from impending doom—a strategy which rarely works.

The Buzz Factor – Check out the industry buzz about the company, the segment in which the company operates in and the competitive landscape. See if they are covered by reputable media sources or if they have one any competitions or awards. A common strategy that some startups use is to make PR releases or pay for favorable coverage. Independent coverage is a good sign that the company is legit and is getting traction. When reading feature articles about the company, look for ranking.  Many publications will provide a listing of the top leaders in the domain. If your company is not in the top list and is just being mentioned using language similar to “also active in this space is…”, this could be a sign that they paid the publisher just to get into print.

Team Makeup – In software more so than in most other engineering disciplines, the human factor and the work environment are critical to success. A salt mine culture and a dysfunctional team are indications that the company will perform poorly. When evaluating the team, inquire about the FTE to contractor ratio. Heavy offshore presence could be an indication that the company is a façade with the bulk of the architecture, development, and engineering work being done offsite/offshore by some outsourced firm. This could a problem if you are under the impression that you are investing in domestic IP and human capital.

Work Culture – The work culture is a good indicator of how functional the organization is. Find out if they are burning the midnight oil every day and if so, why? Are they fixing bugs? Trying to catch-up on backlog features?  Working long hours in a startup is the norm, but doing it for long periods of time could be an indication that they have not yet found their stride. Ask questions like: “What do you love and hate about the company?” or “If you could change three things, what would they be?”

Compensation – This may not be obvious but compensation can teach you a lot about how well the company is doing. Working in a startup requires some financial tradeoffs but the  compensation for the technical team should be within/above the standard industry pay rates. The company should not run like a charity. Did the team get their bonuses last year? Missed yearly bonuses and compensation that is low on cash and high in stock options should raise red flags about how well the company is doing.

Now that you have your finger on the pulse of the organization you are ready to separate the wheat from chaff by identifying the most important takeaways about your target company.

As you complete the two previous DD phases, you will most likely discover that not all of the representations made to you were correct, nor were your original assumptions. The objective of this last exercise is to draw a critical line in the sand that if crossed will result in your walking away from the deal.

The following is my list of eight key assumptions that must pass validation:

1. Platform stability – This covers production matrix such as up-time, downtime, maintenance windows, and singed SLAs. The solution must have published SLA and a historical record of past system shutdowns. All systems go down for one reason or another. It’s important that you understand how frequently their system/sub systems bounce and what the reasons are. The need to babysit the system 24X7 or having a large IT to development ratio can be an indication that the solution is on constant life support.

2. Ease of deployability – This covers questions such as hosting (cloud based vs. hosted), provisioning, and the mechanisms for deployment of new customers and users. When it comes to creating new customer environments, look for manual steps used for copying code, configuring/populating databases, and the usage of script to create work regions. Clearly, any manual process for setting up and boarding customers and the need to manipulate the back-end through manually is a big no-no.

3. Solution scalability – This covers questions regarding number of current transactions per customer, number of customers, daily feed sizes, batch processing schedules, daily feed timeline, and core processing windows. Pay close attention to storage, processing, clustering, and load balancing. Look for obvious signs that the solution will not scale. For example, if the company plans to double its customer base in 12 months, they should already have in place the infrastructure to support such growth. Very few organizations are capable of simultaneously galloping and changing horses mid-stream by making significant alterations to to their storage and load balancing architecture.

4. Maintainability – This covers questions such as production release readiness, customer reporting, and bug tracking. Regardless of how young the company is and their appetite for technology debt, they need to have a functional configuration management, change control, and monitoring capabilities. This doesn’t mean that it’s either HP OpenView or bust. To achieve monitoring, open source tools like Nagios will do. Regardless of the tool, they need to have something in place that is integrated into their solution. Without such controls, they will be flying in the dark, which almost certainly will adversely impact their customers.

5. Disaster recovery, business continuity planning, and availability – This covers questions like how and if the company will recover from various disaster scenarios. What happens if they lose a customer database or the records of important transactions? Is this data being backed up daily? Have they ever attempted to recover from backups? If the company is providing financial services or uses big data, find out how they backup the sensitive information such as PCI data and the terabytes of records on their HDFS.

6. Sophistication of intellectual property – This covers questions regarding the robustness of the algorithms, the structure of the data models, the coupling of the various tiers, the utilization of new and cutting edge frameworks, (i.e. big data components like CPE, queue, plug-ins like R, etc.), and how well everything is mashed together. Remember, just because they use cloud storage/hosting or Hadoop doesn’t mean that their solution can achieve their business objectives or even successfully process large amounts of data.

7. Support for internationalization – This covers questions regarding multi-lingual support, localization, redundant hosting and customer support that follows the sun. Very few startups will be able to fully support internationalization.  If you are planning to offer this solution as part of your international portfolio of products, you will need real internationalization that goes beyond the skin deep ability to customize logos and labels.  Just like in the case of scalability question, if the functionality is not there now, it will require a significant development effort downstream.

8. Security and privacy –  This covers questions regarding authentication, anonymization, encryption, sensitive data storage, data retention, compliance with PCI, FFIEC, etc. Security, due to its nature, is viewed almost universally as overhead and an afterthought. If the platform you are evaluating needs to run silent and deep in hostile waters, you need to make sure that areas such as intrusion detection/prevention, access controls, malware/firewall management, and auditing are up to snuff.  Look for up-to-date security policies, records of ongoing security audits (SAS 70, CISA, etc.), vulnerability assessments reviews, and penetration tests. If the company has no such records on file, this can be a strong indication of poor security planning, which is a ticking liability time bomb.

General Consideration During your Due Diligence
My primary indicator of readiness and prospect for success is the number of customers that currently use the software. Obviously these numbers may vary with the type of the solution but if your investment target has a steady and growing customer base, they have at least survived the valley of death and are for real.  When evaluating the customer base, look for active accounts that use the system regularly.  In many startups, the customers are often made up of relatives/friends and pilot users, although, these types of accounts are important for testing they have little commercial value.

Remember, in the end, it doesn’t matter how compelling the business case may seem, what great technologies they have, or how modular their solution architecture is, without a real customer base, it’s a risky gamble.

A secondary indicator is that of the team and organization. Are you are just buying the software, the team, or the entire package? If you are only interested in the IP, then you will need to identify and secure the architects, lead developers, and core technical team in order to assimilate the technology. On the other hand, if you want the product, then you will need to insure that the organizational structure will be maintained.  This is not an easy thing to do, as often many core team member will cash their chips and move on to pursue other opportunities after the sale of the company.

A third indicator is that of Intellectual Property. You need to carefully address IP questions and determine who owns it, where the inventions come from, who was exposed to the inventions, what are the rights of the FTE/contractors to these ideas, and if there are any invention disclosure forms or patent filings in place.

An in-depth evaluation of the architecture through a code review of the key algorithms, data structure, and framework that form the secret sauce should help answer most of these questions. It is important that you conduct this discovery hands-on by reviewing code and metrics such as code quality, code complexity, and unit test coverage. This is the only way for you to insure that the magic is real.

Executing an effective technology due diligence is more of an art than a science because each software solution you will evaluate is unique. Many early and mid stage startups need to trade off between delivering basic business value and developing a fully mature prime time ready platform. These competing factors make it hard to determine with certainty if a solution has the potential evolve into a commercial success or if it is just being held together with chicken wire and chewing gum.

It is important to approach each discovery phase with a set of simple objectives that are critical for a favorable evaluation of the overall solution. This way, during the evaluation of each key assumption, you will be able to clearly identify the main decision gates and confidently make a go/no-go determination.

© Copyright 2014 Yaacov Apelbaum. All Rights Reserved.


The Time Tunnel & Reciting the Shema in Papua New Guinea

Yaacov Apelbaum-Time Tunnel

Among the most prominent themes in the Hebrew Bible are the concepts of sin, punishment, repentance, and restoration.  Chapter 28 of the book of Deuteronomy, known as the “blessing and curse”, makes it abundantly clear what the rules of the game are. Follow the law and you will enjoy fantastic entrepreneurial success and overflowing prosperity. Disobey it, and you’ll be punished with the worst forms of war, exile, anarchy, and poverty.

The following two promises are good illustrations of the inverse relationship of the biblical punishment and restoration concepts:

Punishment (Deuteronomy 28:64)

…and the LORD shall scatter you among all peoples, from the one end of the earth unto the other end of the earth;

…and there you will serve other gods, which you have not known, nor your fathers, even ones made of wood and stone.

is offset by:

Restoration (Jeremiah 29:14 and Zechariah Chapter 14:9)

… and I will end your captivity, and gather you from all the nations, and from all the places whither I have driven you, said the LORD.

…and the LORD shall be King over all the earth; in that day shall the LORD be One and His name one.

I chose to use the example of the exile from/return to the Promised Land as an illustration because it seems to have been executed with meticulous precision for over 2700 years. Being a software engineer, I can’t help but look at a promise of punishment and restoration that spans such a long period of time and not see a BPEL long-running transaction.

In system design, we use the term ‘long-running transaction’ to describe a job that may need to run for an extended time and survive various failure conditions like system reboots and lack of connectivity.  Another characteristic is that these processes might have long periods of inactivity between consecutive events. This may be because the process is waiting for an external message/event to arrive or occur.

Armed with this useful information, we can begin our historical voyage to examine how this ‘long-running transaction’ has unfolded throughout the centuries:

The Jewish mass exiles begins in 740 BCE. After repeated threats and prophecies foreshadowing impending doom, the Assyrian king Tiglath-Pileser III, arrives to the Northern Kingdom of Israel and exiles the tribes of Reuben, Gad, and the half of the tribe of Manasseh (I Chronicles 5:26).

Yaacov Apelbaum - Sennacherib Prisms In 722 BCE, it is the turn of Samaria, the capital of the Northern Kingdom. After a three year siege, Samaria is captured by Sargon II and Shalmaneser V, each of whom, in turn, proceeds to exile first 27,290 inhabitants of Samaria and then ten of the twelve tribes of Israel. Those ten later came to be known as the Ten Lost Tribes. (2 Kings 17:24).

In 701 BCE, twenty years later, Sennacherib leads a military campaign against Judea, which results in the exile of 200,000 Israelites (2 Kings 18:12 and Taylor Prism).

Yaaacov Apelbaum - Nebchadnezzar Tablet JerusalemNow fast-forward the time machine by 100 years, to 597 BCE.  The Assyrian empire has just been replaced by the Babylonian.  With new regional management comes a new round of exiles. This time it’s king Nebuchadnezzar II who is the divine “messenger”. Jerusalem, the capital of Judea, is put under siege and eventually falls resulting in the destruction of the First Temple in Jerusalem and the exile of 50,000 people to Babylonia (2 Kings 25:21).

By 520 BCE, only 70 years later, the Babylonian empire has gone the way of all empires and the new superpower, Persia, permits the exiles to return to Judea and rebuild the Second Temple.

Yaacov Apelbaum - Antiochus IV EpiphanesIn 334 BCE, the Persian empire finally meets its maker. Judea now falls under the rule of Alexander the Great. In 167 BCE, his successor, the Seleucid king Antiochus IV Epiphanes, pursues a zealous Hellenizing policy against the Jews which leads to the Maccabean Revolt. In the space of three days, 40,000 people are killed in Jerusalem and the same number are exiled and sold into slavery. (2 Maccabees 5:11–14).

By 6 CE, the Seleucid empire bites the dust and Judea became a province of the Roman empire. In 66 CE, due to a combination of religious and political factors, a full Yaacov Apelbaum -Titus Augustus Jewish War Against the Romansblown revolt is launched against Rome. This war, known as the First Jewish–Roman War, lasts for about 7 years and ends in the destruction of Jerusalem and the Second Temple. According to Josephus, around 1,000,000 people are killed and as many as 100,000 are exiled and sold into slavery.

Through their iron fist policy, the Romans keep Judea quiet for about 40 years. Then in 115 CE, the Second Jewish-Roman war breaks out.  Known as the Kitos War, the war lasts for about two years and results in the complete depopulation of many communities and many exiles.

Yaacov Apelbaum - Hadrian Bar Kochvah RebellionPax Romana works for about 15 more years. Then in 132 CE, the emperor Hadrian decides to rename Jerusalem "Aelia Capitolina" and to prohibit circumcision. This leads to the Third Jewish-Roman War, also known as the Bar Kokhba Revolt. The war lasts for 4 years.  The outcome is almost the complete devastation of Jewish life in Judea. According to the Roman historian Cassius Dio, 580,000 Jews were killed and thousands exiled.

In a final attempt to suppress any future Jewish revolts, Hadrian burns the Torah scrolls at the former Temple sanctuary and places two statues there: one of Jupiter and one of himself. Yaacov Apelbaum - Constantius GallusTo eradicate any memory of Judea or Israel, he also wipes the name “Judea” off the official Roman maps and replaces it with “Syria Palaestina” (after the Philistines).

This strategy works for about 120 years. Then in 351 CE, a revolt brakes out against emperor Gallus.  After a short war, Tiberias, Diospolis, and Diocaesarea, the centers of the rebellion, are razed to the ground. Ursicinus, the Roman general in charge, orders thousands to be killed, enslaved, and exiled.

260 years passes, and the empire is now under Byzantine management when a Jewish revolt Yaacov Apelbaum -Heraclius Tremissisbrakes out against emperor Heraclius. The war ends in about 626 and is followed by a wide scale massacre of the Jewish population throughout Jerusalem and Galilee, and the exile of tens of thousands.

By 628 CE, it’s the end of the road for the Byzantine empire. The Jewish population in Judea under Muslims rule continues to shrink for about 400 years and eventually in 1099 CE, culminates in the Crusades during which most of the Jewish population left in the land is either killed or exiled.

Yaacov Apelbaum - Latin Kingdom of Jerusalem Baldwin III.

This pattern continues during the Middle Ages, Renaissance, and up until as late as the 20th century.

Some of the expulsions are massive, such as the one in Spain in 1492 that effects 800,000 people. Others, are smaller and impact a single city or several hundred individuals.  But nevertheless, The Jewish communities everywhere were constantly involuntarily on the move.

A quick historical sampling of European expulsions between 1495-1597 shows 23 such events.

1495 Lithuania
1497 Portugal
1499 Germany
1510 Brandenburg, Germany
1510 Naples  
1514 Strasbourg
1519 Ratisbon [Regensburg in Germany]
1527 Florence
1535 After Spanish troops capture Tunis, all the local Jews are sold into slavery
1540 Naples
1542 Bohemia
1550 Genoa
1551 Bavaria
1551 Pesaro
1559 Austria
1561 Prague
1564 Brest-Litovsk
1567 Würzburg [Bavaria] 
1569 All Papal Territory except Rome and Ancona
1593 Brandenburg, Austria
1597 Cremona
1597 Pavia
1597 Lodi

By 1947, 2700 years have passed since the first Assyrian exile. The original prophesy in Deuteronomy 28:64 of “I’ll scatter you among all people… from the one end of the earth unto the other end of the earth” has now been fulfilled.

So, you are probably thinking to yourself: “This history of the exiles and expulsions is very interesting, but where is the proof of the inverse prophesy?” (Remember? The one about gathering the exiles from the far reaches of the earth and bringing them back to their homeland or the universal recognition of the one nature of God?).

Wonder no more! In what looks like the self-reassembly scene from the Iron Giant, the decedents of the exiles are finally starting to make their way back home. Need some proof?  By 1948, against all odds, the State of Israel is re-established, the land is reclaimed, and Hebrew, as a spoken language is resurrected.  Furthermore, consider the stories of some of the returning exiles, a remote and apparently completely unrelated groups like: Bnei MenasheBene Ephraim, Bene Israel, Pashtun, ye-Ityoppya Ayhudi, Bakwa Dishi, The Lemba people, and Kaifeng. All of these have an oral traditions that claim that they are the descendents of the Judean exiles or the ten lost tribes.

Ok, so what about the universal recognition of the “one nature of God” prophesy? This one takes the cake! Check out the video below, recorded in a remote village in Papua New Guinea. It shows the native community reciting one of the oldest biblical affirmation prayers about the unity of God.

The words for this song come from the text found in Deuteronomy 6:4:

Hear, O Israel: the LORD our God, the LORD is one

For Jews, it is considered the single most important passage in the Hebrew Bible, and it has been recited as part of the daily prayer routine for over 3,000 years, long before the first exile ever took place. 


God bless the people of Papua New Guinea!


© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

The Startup Leap to Success

Yaacov Apelbaum-The Startup Product Leap

One of the most challenging periods for any startup is passing through the “Valley of Death”. During this delicate phase, the organization’s burn rate is high and it has to rapidly achieve the following three goals:

  1. Move from a proof of concept (POC) to a functional commercial product
  2. Reach a cash flow break even
  3. Transition form seed\angel funding to venture capital funding

For startups focusing on the development of SaaS products, this phase also marks an important millstone in the maturity of their product. With increased volume of production users comes stricter SLA’s and the need to implement more advanced operational ability in areas such as: change control, build automation, configuration management, monitoring and data security.

Yaacov Apelbaum-Startup Financing Cycle

If you are managing the technology organization in an early stage startup, you have every reason to be concerned. To the outsider, the success and failure of startups often seems to be shrouded in mystery–part luck part black magic.  But ask a seasoned professional who has successfully gone through the startup meat grinder and he will tell you that success has nothing to do with luck, spells, or incantations.

Having worked with a number of startups, I have come to conclude that the most common reasons for product failure (beyond just not being able to build a viable POC) is the inability to control your product’s stability and scalability.

In the words of Ecclesiastes, there is a time and purpose for everything under heaven.  In the early stages of a startup’s life cycle,  process is negotiable.  Too much process may hinder the speed in which you can build a functional POC.  In later stages, reliable process and procedures (e.g. requirements, QA, unit testing, documentation, build automation, etc., ) are critical. They are the very foundations of any commercial grade product.  Poor quality and performance are self evident and no matter how much marketing spin and management coercion you use, if you are trying to secure an early stage VC funding round, your problems will rapidly surface during the due diligence process.

To avoid the startup blues, keep your eyes on the following areas. Factoring them into your deployment will help you deliver on time and on budget, with the proper scalability and highest quality possible.

Design Artifacts

  1. Before converting your POC to a functional product, take the time to design your core components (i.e. CRM, CMS, DB access, security, API, etc.).  Create a high level design that identifies all major subsystems.  Once you know what they are, zoom into each subsystem and provide a low level design for each these as well.
  2. Resist the temptation to code core functionality before you have a solid and approved scalable architecture (and the documentation for it). 
  3. Let your team review and freely comment about the proposed platform architecture and deployment topology.  Just because a vocal team member has religious technology preferences doesn’t mean that everyone should convert.
  4. No matter how good your technical staff is, when it comes to building complex core functionality (transaction engine, web services API, etc,) don’t give any single individual carte blanche to work in isolation without presenting their design to the entire team.
  5. Document the product as you develop it. Building a complex piece of software without accurate documentation is akin to trying to operate a commercial jet without its flight manual.
  6. To help spread the information and knowledge, establish a company-wide document depository (like a Wiki or SharePoint ) and store all your development and design documents under version control.  Discourage anyone from keeping independent runaway documents of the system.
  7. Maintain an official (and versioned) folder for the platform documentation showing product structure and components, development roadmaps, and technical marketing materials. 

Testing and QA

  1. If you are not writing unit tests you have no way to verify your product’s quality. Relying on QA to find your bugs means that by the time you do (if ever!) it will be too late and expensive to fix them.  Spend a little extra time and write unit tests for every line of code you deploy in production.  When refactoring old code, update the original unit test as well.
  2. Just like most things in life, bugs have a lifecycle, they are born, they live and die.  Effectively tracking them as part of your build and QA process is a prerequisite for their timely resolution.  
  3. If you are discovering a high critical bug count in your “code complete release” (half a percent of source code e.g. 500 bugs for a 100,000 line code base), you may not be production ready.  Stop further deployment and conduct a thorough root cause analysis to understand why you have so many issues. 
  4. If your bug opening/closure rate remains steady (i.e. QA is opening bugs at the same rate development is closing them) and you have reoccurring bug bounces, you may need to reassess the competency of your development resources. This would also be a good time to have a serious heart to heart conversation with the developers responsible for the bugs. Be prepared for some tough HR decisions.

Monitoring and Verification

  1. Just like you wouldn’t drive a car without a functional dashboard, you can’t run quality commercial software without real time visibility into its moving parts.  Implement a monitoring dashboard to track items such as daily builds (and breaks), servers performance, users transactions, DB table space, etc. 
  2. Seeing is believing. Products like Splunk can help you aggregate your operational data.  Once you have this information, show it to your entire team. I personally like to pump it onto a large screen monitor in the development areas so everyone can get a glimpse.

Yaacov Apelbaum-Splunk Monitoring

Image 1: Splunk Dashboard in Action

Security, Scalability and Operations

  1. Unless you are in the snake oil sales business, build your production environment from the get-go for scalability, security, and redundancy.  Don’t look for “bargains” on these technologies, leverage commercial-grade load balancers, firewalls, and backup solutions.
  2. Your production environment is critical to your success, so don’t treat it as a second class citizen or try to manage it with part time resources. As you will quickly discover, a dedicated sys admin and a DBA who know your platform intimately are worth their weight in gold.
  3. You must achieve operational capabilities in build automation, release management, bug tracking, and configuration management before going live.  If you don’t, be prepared to spend most of your productive time fixing boo-boos in the wee hours of the night.
    Implementing many of the above mentioned measures will give you a significant tactical advantage as well as a strategic boost when negotiating with potential VCs.  Having these capabilities on your utility belt will also help you calmly face any future issues as your startup matures.


© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

Only the Racially Pure Need Apply

Yaacov Apelbaum-Fritz Kuhn On February 20, 1939, over 20,000 American supporters of the Nazi party packed Madison Square Garden in New York City. They anxiously awaited the appearance of Fritz Julius Kuhn, the newly anointed Führer of the German-American Bund. The event took place two days before George Washington’s birthday and a 30-foot-portrait of the first president (who was described by Kuhn as the first fascist) hung behind the podium along with Nazi flags and swastikas.

Kuhn entered the arena together with thousands of uniformed Nazi guards. During the rally he and his fiery fellow orators held back no punches, calling President Franklin D. Roosevelt: “Franklin Rosenfeld,” and referring to his New Deal as a "Jew Deal."

Yaacov Apelbaum-George W and Bund Being a creature of the night, Kuhn loved nightclubs, drinking, and the company of women (among them his two mistresses, Virginia Cogswell AKA “The Marrying Georgia Peach–on account of her previous seven husbands”, Florence Camp, Frau Hedwig Munx, and others). Just like many petty dictators, he was pompous, dishonest, idiotic, and didn’t understand his own limitations.

Once during a testimony before the Dies Committee, he was asked by Congressman Starnes if the reason why 23 of 71 Bund units concentrated in and around New York City was because the aircraft and naval manufacturing facilities were handy for sabotage. He replied: "That’s the same thing Lipshitz said. You know who Lipshitz is? That’s Walter Winchell [referring to Winston Churchill]. Lipshitz is his real name."  No one was amused.

Shortly after his rock concert-like appearance in Madison Square Garden, New York city’s mayor, La Guardia, who was fed up with the constant anti-Semitic and anti-American agitation, started an Al Capone-style financial investigation of the Bund’s taxes.

When asked about his relationship to Florence Camp during his trial, Kuhn denied that he had asked her to marry him and noted that Mrs. Camp was too much of a lady to accept a proposal after just a few days’ acquaintance. Herman McCarthy (the prosecutor) whipped out a Kuhn letter and read it aloud:

"Florence : I am terrible in love with you. I beg you to become my beloved wife. I will always be true to you. . . ."

In another letter to Florence he said that he loved her with his “whole soul and body and was about to have [his] teeth fixed.”

In the course of the trial, it was established that Kuhn had pilfered $14,548 from his organization ($717.02 of it having been spent on moving expenses for Mrs. Camp). Kuhn was swiftly convicted on charges of embezzlement, grand larceny, and forgery and was first sent to Sing Sing Prison.  After the war, he was deported to Germany, where he managed to get into trouble again.

Yaacov Apelbaum-The Escape of Fritz KuhnIn 1949 when he again stood trial in front of a Munich court this time on charges of escaping from jail and being a major Nazi organizer, he claimed that the Bund was strictly "an American patriotic organization," that he had used the swastika only because it was "an old American Indian design," and that he had patterned the Bund’s uniforms after the US National Guard, not  the SS. As for his 1944 meeting with Hitler he said: 

"It was purely a social call. If I went to England today, I would naturally like to call on King George." 

When the US. entered the war, whatever was left of the German-American Bund organization quickly disintegrated, however, that didn’t spell the end of Nazi activity in America. Another high profile organization waiting in the wings was the Steuben Society. In comparison to the Bund which was composed of common National Socialist riff raff, the Steuben Society represented the cream of the crop to the US Nazi aristocracy.

Although Steuben Society members avoided public Nazi displays such as hailing Hitler, the differences between the two organizations were only skin deep. When it came to hard core issues such Nazi ideology, they were indistinguishable.

While visiting the reception room of the Steuben Society in New York, John Roy Carlson observed:

“One could find a large American flag standing in one corner. On the walls were pictures of Von Steuben, Washington, and Lincoln, The Pledge to the Flag and the Bill of Rights hung framed between them. There was also no lack of red-white-and-blue. Patriotism oozed from every crevice in the room.”

True to its nature, the Society published “The Steuben News" a newspaper for Patriotic Americans which described itself as:

Yaacov Apelbaum-The Steuben News . . . a patriotic, civic and educational political society endeavoring to awaken in the hearts and minds of American citizens of German extraction the necessity for taking a more active part and interest in the political affairs of our great country.

Its program demanded "strict discipline" on the part of its members, and rejected "persons who are shifters and trimmers, or who are known to possess no race pride." The Steuben Society strongly emphasized Racial (Aryan) consciousness and political objectives.

In his 1943 investigative book Under Cover, Carlson wrote:

“…The Steuben News reprinted articles from the pro-Fascist Italian daily, Il Progress Halo-Americana. It recommended books by the notorious Ausland Institute and ran many articles by Nazi agents. The Steuben News praised as "extraordinary and valuable" the book Scarlet Fingers published by Flanders Hall, the propaganda mill financed by Nazi agent George Sylvester Viereck. The Steuben News followed the accepted party line of pro-Nazi isolationists. It headlined the speeches of Lindbergh. It championed the late senator Ernest Lundeen-some of whose speeches were written by Nazi agent George Sylvester Viereck-and on one occasion devoted eleven columns to one of his defeatist speeches.

It reprinted from Social Justice and The Herald, American Fascist weekly. It ran large advertisements for the America First Committee, reprinted its bulletins and urged its members to support it financially. The Steuben Society fought desperately all measures to arm those European Democracies which resisted Hitler’s brutality. And it also quoted liberally from the New York Enquirer, published by William Griffin, who was later shown to have associated with Viereck.”

Now, you’re probably thinking: “This is a fascinating piece of history, but what’s the relevance of all of this 1939 Nazi stuff to our current 21st century jet-set life style?” Well, wonder no more.

This past Sunday morning on our way out of our local diner, I caught sight of the newspaper stand in the entrance vestibule.  I usually don’t read printed media, but the name of the paper and the motto “A Newspaper for Americans” caught my attention. Curious about how the Steuben Society’s defines “American,” I picked up my free copy and read on. 

Yaacov Apelbaum-Steuben News

At the top of the cover page on each side of the title “The Steuben News” were the mission statements: (1) United for Common Interests and Common Needs” and (2) DUTY, JUSTICE, TOLERANCE, CHARITY.

I flipped through and read some of the articles. There was an announcement of a presidential proclamation regarding the German-American Day, a story about the treaty between German settlers of Texas and the native Comanche Indians. My first impression was that it all seemed rather banal. Then I got the last page. Under the calendar of events, I ran into some terminologies like “event sponsored by Unit #998” and “contact Brother Erick or Sister Hildegard.” That seemed a bit cryptic and militant. At the bottom of the page I saw the membership form which prompted an unexpected double-take.

The membership form, unlike any other application I have ever seen,  had questions about the nationality of the applicant’s father and mother,  political affiliation, and—most surprising of all—about naturalization. For some reason, the Steuben Society (acting in the capacity of a quasi-government organization?) will only issue membership cards after careful evaluation of the applicant’s naturalization certificate, which includes scrutiny of the certificate number and place of origin. (I’m kind of curious to know who at the INS helps them validate these applications.)

Yaacov Apelbaum-Are You Naturalized

From what I can tell, this membership application has remained consistent over the years. After conducting a quick search on-line for similar historical documents, I found one for the Silver Shirts, and as you can see from the contents, not much has changed in terms of drilling down to pedigree and other über eugenics.

Yaacov Apelbaum-Silver ShirtWhen, I checked out the Steuben Society’s website for the name and location of the chapter nearest me, I discovered that they are all named after some distinguished German American figure. I was hoping to find a chapter honoring the likes of  von Stauffenberg, but alas, no such luck.

I am not sure what to make of all this. I hold German culture,  ingenuity, work ethics, and organization in the highest esteem.  I’m an avid admirer of Handel’s music and Nietzsche’s, Kant’s, Goethe’s, and Leibniz’s  writings.  My family originated from Germany and in my travels there I have found most German people to be kind, polite, friendly, and exceedingly intelligent.

On one hand, it’s laughable that anyone would be willing to complete an application detailing his mother’s nationality or his naturalization number in order to join a civic organization. On the other hand it’s really disturbing that in 2010—the age of the internet—a nationwide fraternity that draws its philosophy from one of humankind’s darkest moments, continues to operate in the mainstream with apparently unrestricted access to leading politicians and public figures.

If you are considering joining an organization such as this, take a breather and dedicate some time to learning the German language, literature, philosophy, and music instead.  You will discover that the richness of Germanic culture has a lot to do with individuality and little with purity of blood.   

Et si omnes ego non

© Copyright 2009 Yaacov Apelbaum All Rights Reserved.

An Afternoon with a Fraudster

Yaacov Apelbaum-The Fraudster

Your Friends at “Account Services”

Having spent a significant amount of time developing fraud detection algorithms and security applications, I have become accustomed to envisioning the common would-be cyber attacker as an inanimate abstract entity completely devoid of human traits; a mere abstraction, a stick figure in my UML and Test Cases. This sterile view of mine however, changed recently when I actually got a chance to spend some time one-on-one with a flesh and blood fraudster.

It started with a seemingly innocuous automated call from “Account Services”. The message informed me that I qualified for a limited time offer to lower my monthly credit card payments. I ignored that first call but shortly afterwards I received a second one. This time I opted to accept the call and was routed to a live representative. I told her that I was not interested in their services and did not want to be contact by them again.

At the tail end of the conversation as I was about to hang up, I inquired about how they got my phone number (it’s both unlisted and on the DNC registry) and to my surprise, the representative said that it came from my bank. When I asked which one, she became evasive, telling me that her company serviced all major banks. That was the moment I realized that I was the target of Credit Card fraud actively in progress.

Suddenly, my stick figure cyber attacker was no longer virtual. Instead, it became a living and breathing human being, an arm’s reach away on the other side of the line. This, I realized, was a rare opportunity to interview an attacker. I asked the individual to call me back on another line and when the phone rang a few seconds later, I raised my foreign accent by a notch, plugged the phone into my MP3 player and hit the Record button.

The representative identified herself as “Michelle. She sounded young, in her twenties. She spoke in a monotonous but confident voice, clearly a veteran of many exploits. The sales pitch was entirely script-based. She inquired about my current balance and asked if I had any interest in lowering my monthly payments. When I said, “I sure do,” she asked me for my bank and credit card information in order to “qualify” me. At that point we began a stubborn cat and mouse game where I was trying to get more information about her whereabouts and identity (real-phone number, e-mail, web address) while she was trying to get my bank and account information. This lasted for approximately 10 minutes all told.

It was only after I played back the recording and listened to it several times that I realized how sophisticated the operation was (you can hear the recording below).

The perpetrators of this scam had thought of the minutest details and prepared for every scenario. Some of the more interesting elements of the call included:

  1. Psychological Usage of Ambient Sound—During the duration of the call, I could hear incoming phone calls and chatter in the background. This recording simulating a response hotline was designed to create the illusion that I was talking to a busy call center. The objective of this subliminal messaging is similar to that used during TV fundraisers where operators are filmed sitting behind desks of ringing phones. All of it is meant to convince us that many others have already taken the plunge and that the water is "fine”.
  2. Call Traceability and Legitimacy—When I asked the rep where her call center was located she successfully identified the state that corresponded to the area code that appeared on my caller ID. I decided to test the number from my cell phone. The phone rang several times but when it was finally answered, I was routed to voicemail and encouraged to leave a message. The fact that the number yielded a response at all certainly made it appear legitimate.
  3. Well Scripted Dialogs—During the conversation, the rep responded in a consistent manner to my questions, reminding me (4 times) that I was being given the opportunity to lower my monthly interest payments. When I voiced my concern about the possibility that this call could be fraudulent, she responded calmly by stating (4 times) that even if this was the case, I would be covered for any losses by my credit card issuer as well as the Federal Consumer Protection Act.
  4. Plausibility—When I asked if I could call her back on another line to verify her number, she explained that hers was an outbound only call center. She also insisted that this was merely a screening call and that I was only a step away from being transferred to an account executive who would be happy to provide me with complete contact information.
  5. Professional Composure and Manners—Even though I asked her the same questions a number of times, she remained polite and composed, always maintaining a businesslike demeanor and projecting a image of a legitimate customer service representative.
  6. Effective Use of Higher Authority—When I insisted that not getting a manned phone number for the representative would be a deal breaker for me, she finally offered to transfer me to her manager. I was placed on hold (listening to Beethoven’s Für Elise) and was soon connected to another individual who identified herself as “LaFonda”, the floor supervisor. She sounded a bit older and more mature. She reiterated the previous sales pitch. When I finally told her that without being able to validate their authenticity I would not be able to give her my credit card number, she gave me the impression that they might deviate from their ‘account information first’ protocol. I was placed on hold again but shortly afterwards my original sales associate was back pitching the same story all over again. Finally, after one last failed sales attempt she quickly wrapped up the call and hung up.

Even though the call only lasted a relatively short time, I could not have wished for a better and more illuminating lesson. My mental image of the on-line fraudster has changed irrevocably. Whereas before I viewed fraud as an opportunistic low tech effort executed by crafty individuals, I now view it as a commercial enterprise, in many ways similar to a legitimate telemarketing niche industry. It employs a well trained workforce, cutting edge BI, telecom technology and a large database of would-be "customers".

In retrospect, the whole experience was both sobering and frustrating. It was sobering because I finally realized that at its core, fraud is propagated via subtle means and recognizing it requires the aggregation of many nuances which individually may appear inconsequential (note that until its collapse, each individual component of Bernard Madoff’s asset management operation appeared to be entirely legitimate). In my case, the red flag went up because of my experience in the financial industry. As a rule, the association between a specific “Credit Card Service” organization and all commercial banks is unlikely. For another individual however, this certainly could have been a plausible explanation and this applies to everything else that was said during the conversation.

The frustration, on the other hand, comes from the realization that my current toolbox of risk analysis and fraud detection routines (which are primarily based on triggers like transaction frequency, amount, location and history) cannot independently identify this type of fraud and will require for at least the foreseeable future some supplemental human supervision.

© Copyright 2009 Yaacov Apelbaum All Rights Reserved.