Tokens of Distrust
It was on a starless March night,
The spear phishers went out for bite.
Through a zero day vulnerability,
They breached RSA’s network security.
A Trojan attached to an email transmission,
Gave the attackers remote access permission.
Deep into the corporate systems they dove,
Collecting the SecureID key seeds treasure trove.
The theft effected over forty million tokens,
Transparency failed and trust was broken.
A few weeks followed and on a moonless May night,
The spear fishers returned with a renewed appetite.
Over the internet via secure VPN and a forged key,
They breached Lockheed’s defenses and Pwned ’their IP.
Ties That Bind
Identity, how do I bind thee to an object? Let me recount the days.
In the days of Spring, by a secret.
In the days of Summer, by a token.
In the days of Fall, by who you are.
In the days of Winter, by seasons’ past and where you’ve been.
A little shiftless man named Phil,
got a CISO gig through a shady deal.
Clueless about APT threats,
He managed upwards like a rat.
Pen tests and audits took a back seat,
To what the cafeteria was serving to eat.
When the company was finally breached by a hack,
He said “C’est la vie! The insurance will cover our back.”
IP traffic flood,
Malignant packets rush-in,
Snort calmly says: “Halt!”
Maybe you are thinking about buying a new technology platform or investing in a software startup. Following industry practices, you will likely conduct some form of due diligence before you make your big move. This may include interviewing members of the management, technology and finance teams. You may also conduct operational audits, review sales figures, talk to customers, and check for references.
All advisable but in the end, you will still be left with a certain amount of nagging doubt. After all, how do you really know what this company’s true technology abilities are? How can you tell with a high degree of certainty that you are not buying the Brooklyn Bridge equivalent of some useless/over-hyped software? In today’s frenzied Internet of Things, mobile and Big Data buzz-ridden world, sometimes it seems as if the sky is the limit. To the uninitiated, it is exceedingly difficult to tell the difference between a solid early stage software idea and a useless concept professing to be the next big, anti-gravity SaaS solution.
I know. You are probably asking yourself: how difficult can it be? After all there are numerous simplified due diligence guides that answers questions like:
● Does the company really own its supposed product?
● Is the technology integrated/constructed in the right way?
● Can their technology scale?
Unfortunately, when you are evaluating a technology potential, you may find that the answers to such questions are fuzzy and not always easily discernable . So before you make your investment decision based on some generic checklist, you may want to consider the following tale about the rise and fall of a flying super hero in tights.
In 2010, following the meteoric success of the Spider-Man movie franchise—which grossed over $2.5 billion worldwide—a stage adaptation entitled “Spider-Man: Turn Off the Dark” arrived to Broadway. The investors spared neither expenses nor talent in pouring over $75 million into the production in hopes of recreating the movie magic and revenue.
To stay true to Spider-Man’s legacy, the play executed some complex aerobatics sequences and flight scenes across the stage. These stunts quickly gained notoriety as the show became plagued by accidents.
Some of the more noteworthy injuries included:
● Stunt double Kevin Aubin broke both wrists when he was catapulted from one end
of the stage to the other
● Brandon Rubendall broke a toe that same month doing the same stunt as Aubin
● Natalie Mendoza, who played villain Arachne, suffered a concussion when she was struck in
the head with a piece of equipment
● Carpio, Mendoza’s replacement, suffered a neck injury after a battle scene with
● Stuntman Christopher Tierney fell 30 feet into the orchestra pit suffering a fractured skull, a
fractured shoulder blade, four broken ribs, and three broken vertebrae
● Daniel Curry, a stunt double, got his right foot stuck in a stage lift and then a trapdoor
closed on the foot, breaking the foot and both of his legs, necessitating amputations
This reads more like an account from the trenches of Verdun than a Broadway musical. Despite the carnage, the performances went on with regular venue changes and constant retooling of the storyline and musical score.
Even negative press reviews such as the “Pigs Will Fly Before Spider-Man Recoups $65 Million Costs” could not stop the show.
Finally last month, the producers announced that they plan to end the production in January 2014, the main reasons being falling ticket sales and—not surprisingly—the inability to get injury insurance for the cast.
In the end, the show will have run for over three years and will have lost an estimated $60 million.
So, what went wrong? Why did life fail to imitate art? It seems that on the live stage, the same stunts that were so easy to achieve in virtual CGI failed miserably when ported to the physical world. Why wasn’t it obvious from the start that the Spider-Man storyline could only work in the pages of comics and on the silver screen?
The investors behind the Broadway adaptation were seasoned entertainment entrepreneurs. Before committing funds to the project, they conducted their due diligence and found the venture to be worthy. Yet over a period of 3 years and despite watching repeating cycles of misfortune, they failed to pull the plug. Apparently, hope springs eternal—at least in the investor’s breast. Sometimes, even though red flags may be staring you right in the face, you can still miss all of the warning signs.
Image 1: Spider-Man Planned vs. Actual
Over the years, I have conducted due diligence on various software partnerships, acquisitions, and investment opportunities. It turns out that questions like: ‘how scalable/portable is this solution?’, or ‘how valuable is the code?’ are not only difficult to answer but often irrelevant.
And just like in the example of the Spider-Man fiasco, even seasoned professionals can fall victim to a well rehearsed pitch presented by a charismatic team of snake oil salesman who can sell you dehydrated water without even blinking.
In many ways, evaluating an investment opportunity in software is like a game of cat and mouse. Your evaluation will involve constant pursuit, near captures, and repeated escapes. You will have to sift through piles of partial facts, exaggerations, and in some cases even deliberate misinformation.
This is to be expected. No cause for alarm though. Here is a three phase approach to conducting due diligence effective enough to help strip the thin veneer of pretense so that you can get deeper insight into how your potential acquisition functions and what its possible soft spots are.
Before you start probing any soft spots, though, you will need to get the regular DD action items out of the way. Conduct some background research and get Intel on the following:
● Litigation (are the company and/or it’s principals in court for any reason?)
● Costs to operate the business for the next 12 months based on current burn down rate
● 3rd party licenses and vendor agreements (both, in terms of income and expense)
● Customer base, future growth projections, and teaming agreements
● Forecasted capital investments (what are the costs of boarding one new customer?)
Now that you have the basics you can proceed to look for chinks in the armor. Schedule some face time with the technology team, including: architects, operations, IT, development, QA, etc. It is important that you conduct both group and personal interviews with these individuals because the group dynamics will effect the detail and quality of the answers you get.
The topics that I find to be the most illuminating include:
Management Pedigree – Find out if the the leadership team has prior successful entrepreneurial experience. Take the time to check them out on-line before meeting them face to face. (LinkedIn is a great source for this.) Each technical leader should have at least five to seven years of “specific and proven” domain experience in the areas that the company is trying to innovate (i.e. mobile, API, Big Data analytics, cloud hosting, scalability, high volume transactional systems, and security). A lack of such experience will dramatically decrease the chances of their success because they will have to learn on the job and this will undoubtedly be error prone and costly.
Also, look into the tenure of the key members on the technical team. Has the CTO, VP of engineering, Director of development, etc. been with the company from the get go? Is there rapid turnover in any of these key positions? A revolving door syndrome could be an indication that the company failed to mature their technology and is trying to bridge the gap by searching for “the one” who will save them from impending doom—a strategy which rarely works.
The Buzz Factor – Check out the industry buzz about the company, the segment in which the company operates in and the competitive landscape. See if they are covered by reputable national media. A common strategy that some startups use is to make PR releases or pay for favorable coverage. Independent coverage is a good sign that the company is legit and is getting traction. When reading feature articles about the company, look for ranking. Many publications will provide a listing of the top leaders in the domain. If your company is not in the top list and is just being mentioned using language similar to “also active in this space is…”, this could be a sign that they paid the publisher just to get into print.
Team Makeup – In software more so than in most other engineering disciplines, the human factor and the work environment are critical to success. A salt mine culture and a dysfunctional team are indications that the company will perform poorly. When evaluating the team, inquire about the FTE to contractor ratio. Heavy offshore presence could be an indication that the company is a façade with the bulk of the architecture, development, and engineering work being done offsite/offshore by some outsourced firm. This could a problem if you are under the impression that you are investing in domestic IP and human capital.
Work Culture – The work culture is a good indicator of how functional the organization is. Find out if they are burning the midnight oil every day and if so, why? Are they fixing bugs? Trying to catch-up on backlog features? Working long hours in a startup is the norm, but doing it for long periods of time could be an indication that they have not yet found their stride. Ask questions like: “What do you love and hate about the company?” or “If you could change three things, what would they be?”
Compensation – This may not be obvious but compensation can teach you a lot about how well the company is doing. Working in a startup requires some financial tradeoffs but the compensation for the technical team should be within/above the standard industry pay rates. The company should not run like a charity. Did the team get their bonuses last year? Missed yearly bonuses and compensation that is low on cash and high in stock options should raise red flags about how well the company is doing.
Now that you have your finger on the pulse of the organization you are ready to separate the wheat from chaff by identifying the most important takeaways about your target company.
As you complete the two previous DD phases, you will most likely discover that not all of the representations made to you were correct, nor were your original assumptions. The objective of this last exercise is to draw a critical line in the sand that if crossed will result in your walking away from the deal.
The following is my list of eight key assumptions that must pass validation:
1. Platform stability – This covers production matrix such as up-time, downtime, maintenance windows, and singed SLAs. The solution must have published SLA and a historical record of past system shutdowns. All systems go down for one reason or another. It’s important that you understand how frequently their system/sub systems bounce and what the reasons are. The need to babysit the system 24X7 or having a large IT to development ration can be an indication that the solution is on constant life support.
2. Ease of deployability – This covers questions such as hosting (cloud based vs. hosted), provisioning, and the mechanisms for deployment of new customers and users. When it comes to creating new customer environments, look for manual steps used for copying code, configuring/populating databases, and the usage of script to create work regions. Clearly, any manual process for setting up and boarding customers and the need to manipulate the back-end through manualy is a big no-no.
3. Solution scalability – This covers questions regarding number of current transactions per customer, number of customers, daily feed sizes, batch processing schedules, daily feed timeline, and core processing windows. Pay close attention to storage, processing, clustering, and load balancing. Look for obvious signs that the solution will not scale. For example, if the company plans to double its customer base in 12 months, they should already have in place the infrastructure to support such growth. Very few organizations are capable of simultaneously galloping and changing horses mid-stream by making significant alterations to to their storage and load balancing architecture.
4. Maintainability – This covers questions such as production release readiness, customer reporting, and bug tracking. Regardless of how young the company is and their appetite for technology debt, they need to have a functional configuration management, change control, and monitoring capabilities. This doesn’t mean that it’s either HP OpenView or bust. To achieve monitoring, open source tools like Nagios will do. Regardless of the tool, they need to have something in place that is integrated into their solution. Without such controls, they will be flying in the dark, which almost certainly will adversely impact their customers.
5. Disaster recovery, business continuity planning, and availability – This covers questions like how and if the company will recover from various disaster scenarios. What happens if they lose a customer database or the records of important transactions? Is this data being backed up daily? Have they ever attempted to recover from backups? If the company is providing financial services or uses big data, find out how they backup the sensitive information such as PCI data and the terabytes of records on their HDFS.
6. Sophistication of intellectual property – This covers questions regarding the robustness of the algorithms, the structure of the data models, the coupling of the various tiers, the utilization of new and cutting edge frameworks, (i.e. big data components like CPE, queue, plug-ins like R, etc.), and how well everything is mashed together. Remember, just because they use cloud storage/hosting or Hadoop doesn’t mean that their solution can achieve their business objectives or even successfully process large amounts of data.
7. Support for internationalization – This covers questions regarding multi-lingual support, localization, redundant hosting and customer support that follows the sun. Very few startups will be able to fully support internationalization. If you are planning to offer this solution as part of your international portfolio of products, you will need real internationalization that goes beyond the skin deep ability to customize logos and labels. Just like in the case of scalability question, if the functionality is not there now, it will require a significant development effort downstream.
8. Security and privacy – This covers questions regarding authentication, anonymization, encryption, sensitive data storage, data retention, compliance with PCI, FFIEC, etc. Security, due to its nature, is viewed almost universally as overhead and an afterthought. If the platform you are evaluating needs to run silent and deep in hostile waters, you need to make sure that areas such as intrusion detection/prevention, access controls, malware/firewall management, and auditing are up to snuff. Look for up-to-date security policies, records of ongoing security audits (SAS 70, CISA, etc.), vulnerability assessments reviews, and penetration tests. If the company has no such records on file, this can be a strong indication of poor security planning, which is a ticking liability time bomb.
General Consideration During your Due Diligence
My primary indicator of readiness and prospect for success is the number of customers that currently use the software. Obviously these numbers may vary with the type of the solution but if your investment target has a steady and growing customer base, they have at least survived the valley of death and are for real. When evaluating the customer base, look for active accounts that use the system regularly. In many startups, the customers are often made up of relatives/friends and pilot users, although, these types of accounts are important for testing they have little commercial value.
Remember, in the end, it doesn’t matter how compelling the business case may seem, what great technologies they have, or how modular their solution architecture is, without a real customer base, it’s a risky gamble.
A secondary indicator is that of the team and organization. Are you are just buying the software, the team, or the entire package? If you are only interested in the IP, then you will need to identify and secure the architects, lead developers, and core technical team in order to assimilate the technology. On the other hand, if you want the product, then you will need to insure that the organizational structure will be maintained. This is not an easy thing to do, as often many core team member will cash their chips and move on to pursue other opportunities after the sale of the company.
A third indicator is that of Intellectual Property. You need to carefully address IP questions and determine who owns it, where the inventions come from, who was exposed to the inventions, what are the rights of the FTE/contractors to these ideas, and if there are any invention disclosure forms or patent filings in place.
An in-depth evaluation of the architecture through a code review of the key algorithms, data structure, and framework that form the secret sauce should help answer most of these questions. It is important that you conduct this discovery hands-on by reviewing code and metrics such as code quality, code complexity, and unit test coverage. This is the only way for you to insure that the magic is real.
Executing an effective technology due diligence is more of an art than a science because each software solution you will evaluate is unique. Many early and mid stage startups need to trade off between delivering basic business value and developing a fully mature prime time ready platform. These competing factors make it hard to determine with certainty if a solution has the potential evolve into a commercial success or if it is just being held together with chicken wire and chewing gum.
It is important to approach each discovery phase with a set of simple objectives that are critical for a favorable evaluation of the overall solution. This way, during the evaluation of each key assumption, you will be able to clearly identify the main decision gates and confidently make a go/no-go determination.
© Copyright 2014 Yaacov Apelbaum. All Rights Reserved.