The Far East is a wonderous place. Over the past two years, I’ve spent a lot time there on business. When I’m not burning the midnight oil at work, I try to steal some time off on the weekends to go hiking.
I recently stopped in Japan on the advice of a friend and visited Narita’s Shinshō-ji temple.
Modern Japan is pretty industrialized by now, so it’s hard to find locations that confirm the romantic silk paintings that most of us associate with old Japan. Shinshō-ji, however, is a bona fide and beautiful vestige of the past.
The temple is located about 20 minutes by train from the airport,so if you are between flights and have a few hours to spare, store your luggage at the airport and check out this place. It is the poster child of pastoral Japan: a small village with traditional stores and houses, beautiful gardens, and breathtaking architecture.
© Copyright 2014 Yaacov Apelbaum All Rights Reserved.
Second day of the WCS and we have been demoing the City Falcon, an Unmanned Aerial Vehicles platform designed especially to operate in crowded urban environments.
Standard features include:
- Low RPM brushless motors
- All weather operational capability
- Fully autonomous flight and support for complex orchestrations
- Customization for multi radio frequencies
- Fire resistant design
- Long range flights (over 5 miles)
In addition to all of the above, the UAV also supports specialty custom payloads like environmental, chemical sensors, and multiple types of cameras.
Operational demonstrations included:
- Flight through high heat sources (Oil tank fire of over 450º F)
- Operating in high wind and high turbulence environments
- Evaluation of crowd formation and people flow
- Autonomous flights path and homing
- Stalking multiple targets of interest
- Patrolling and inspections of a chemical plant and hazardous substances
© Copyright 2014 Yaacov Apelbaum. All Rights Reserved.
First day of the WCS—against all odds and after two months of grueling development work, an impossible deadline, and 15 hour days—we delivered and deployed the world’s first standalone, wearable face recognition system.
It is my humble hope that this small contribution of ours will help to make our cities, communities, and public gatherings safer.
© Copyright 2014 Yaacov Apelbaum All Rights Reserved.
Tokens of Distrust
It was on a starless March night,
The spear phishers went out for bite.
Through a zero day vulnerability,
They breached RSA’s network security.
A Trojan attached to an email transmission,
Gave the attackers remote access permission.
Deep into the corporate systems they dove,
Collecting the SecureID key seeds treasure trove.
The theft effected over forty million tokens,
Transparency failed and trust was broken.
A few weeks followed and on a moonless May night,
The spear fishers returned with a renewed appetite.
Over the internet via secure VPN and a forged key,
They breached Lockheed’s defenses and Pwned ’their IP.
Ties That Bind
Identity, how do I bind thee to an object? Let me recount the days.
In the days of Spring, by a secret.
In the days of Summer, by a token.
In the days of Fall, by who you are.
In the days of Winter, by seasons’ past and where you’ve been.
A little shiftless man named Phil,
got a CISO gig through a shady deal.
Clueless about APT threats,
He managed upwards like a rat.
Pen tests and audits took a back seat,
To what the cafeteria was serving to eat.
When the company was finally breached by a hack,
He said “C’est la vie! The insurance will cover our back.”
IP traffic flood,
Malignant packets rush-in,
Snort calmly says: “Halt!”
Maybe you are thinking about buying a new technology platform or investing in a software startup. Following industry practices, you will likely conduct some form of due diligence before you make your big move. This may include interviewing members of the management, technology and finance teams. You may also conduct operational audits, review sales figures, talk to customers, and check for references.
All advisable but in the end, you will still be left with a certain amount of nagging doubt. After all, how do you really know what this company’s true technology abilities are? How can you tell with a high degree of certainty that you are not buying the Brooklyn Bridge equivalent of some useless/over-hyped software? In today’s frenzied Internet of Things, mobile and Big Data buzz-ridden world, sometimes it seems as if the sky is the limit. To the uninitiated, it is exceedingly difficult to tell the difference between a solid early stage software idea and a useless concept professing to be the next big, anti-gravity SaaS solution.
I know. You are probably asking yourself: how difficult can it be? After all there are numerous simplified due diligence guides that answers questions like:
● Does the company really own its supposed product?
● Is the technology integrated/constructed in the right way?
● Can their technology scale?
Unfortunately, when you are evaluating a technology potential, you may find that the answers to such questions are fuzzy and not always easily discernable . So before you make your investment decision based on some generic checklist, you may want to consider the following tale about the rise and fall of a flying super hero in tights.
In 2010, following the meteoric success of the Spider-Man movie franchise—which grossed over $2.5 billion worldwide—a stage adaptation entitled “Spider-Man: Turn Off the Dark” arrived to Broadway. The investors spared neither expenses nor talent in pouring over $75 million into the production in hopes of recreating the movie magic and revenue.
To stay true to Spider-Man’s legacy, the play executed some complex aerobatics sequences and flight scenes across the stage. These stunts quickly gained notoriety as the show became plagued by accidents.
Some of the more noteworthy injuries included:
● Stunt double Kevin Aubin broke both wrists when he was catapulted from one end
of the stage to the other
● Brandon Rubendall broke a toe that same month doing the same stunt as Aubin
● Natalie Mendoza, who played villain Arachne, suffered a concussion when she was struck in
the head with a piece of equipment
● Carpio, Mendoza’s replacement, suffered a neck injury after a battle scene with
● Stuntman Christopher Tierney fell 30 feet into the orchestra pit suffering a fractured skull, a
fractured shoulder blade, four broken ribs, and three broken vertebrae
● Daniel Curry, a stunt double, got his right foot stuck in a stage lift and then a trapdoor
closed on the foot, breaking the foot and both of his legs, necessitating amputations
This reads more like an account from the trenches of Verdun than a Broadway musical. Despite the carnage, the performances went on with regular venue changes and constant retooling of the storyline and musical score.
Even negative press reviews such as the “Pigs Will Fly Before Spider-Man Recoups $65 Million Costs” could not stop the show.
Finally last month, the producers announced that they plan to end the production in January 2014, the main reasons being falling ticket sales and—not surprisingly—the inability to get injury insurance for the cast.
In the end, the show will have run for over three years and will have lost an estimated $60 million.
So, what went wrong? Why did life fail to imitate art? It seems that on the live stage, the same stunts that were so easy to achieve in virtual CGI failed miserably when ported to the physical world. Why wasn’t it obvious from the start that the Spider-Man storyline could only work in the pages of comics and on the silver screen?
The investors behind the Broadway adaptation were seasoned entertainment entrepreneurs. Before committing funds to the project, they conducted their due diligence and found the venture to be worthy. Yet over a period of 3 years and despite watching repeating cycles of misfortune, they failed to pull the plug. Apparently, hope springs eternal—at least in the investor’s breast. Sometimes, even though red flags may be staring you right in the face, you can still miss all of the warning signs.
Image 1: Spider-Man Planned vs. Actual
Over the years, I have conducted due diligence on various software partnerships, acquisitions, and investment opportunities. It turns out that questions like: ‘how scalable/portable is this solution?’, or ‘how valuable is the code?’ are not only difficult to answer but often irrelevant.
And just like in the example of the Spider-Man fiasco, even seasoned professionals can fall victim to a well rehearsed pitch presented by a charismatic team of snake oil salesman who can sell you dehydrated water without even blinking.
In many ways, evaluating an investment opportunity in software is like a game of cat and mouse. Your evaluation will involve constant pursuit, near captures, and repeated escapes. You will have to sift through piles of partial facts, exaggerations, and in some cases even deliberate misinformation.
This is to be expected. No cause for alarm though. Here is a three phase approach to conducting due diligence effective enough to help strip the thin veneer of pretense so that you can get deeper insight into how your potential acquisition functions and what its possible soft spots are.
Before you start probing any soft spots, though, you will need to get the regular DD action items out of the way. Conduct some background research and get Intel on the following:
● Litigation (are the company and/or it’s principals in court for any reason?)
● Costs to operate the business for the next 12 months based on current burn down rate
● 3rd party licenses and vendor agreements (both, in terms of income and expense)
● Customer base, future growth projections, and teaming agreements
● Forecasted capital investments (what are the costs of boarding one new customer?)
Now that you have the basics you can proceed to look for chinks in the armor. Schedule some face time with the technology team, including: architects, operations, IT, development, QA, etc. It is important that you conduct both group and personal interviews with these individuals because the group dynamics will effect the detail and quality of the answers you get.
The topics that I find to be the most illuminating include:
Management Pedigree – Find out if the the leadership team has prior successful entrepreneurial experience. Take the time to check them out on-line before meeting them face to face. (LinkedIn is a great source for this.) Each technical leader should have at least five to seven years of “specific and proven” domain experience in the areas that the company is trying to innovate (i.e. mobile, API, Big Data analytics, cloud hosting, scalability, high volume transactional systems, and security). A lack of such experience will dramatically decrease the chances of their success because they will have to learn on the job and this will undoubtedly be error prone and costly.
Also, look into the tenure of the key members on the technical team. Has the CTO, VP of engineering, Director of development, etc. been with the company from the get go? Is there rapid turnover in any of these key positions? A revolving door syndrome could be an indication that the company failed to mature their technology and is trying to bridge the gap by searching for “the one” who will save them from impending doom—a strategy which rarely works.
The Buzz Factor – Check out the industry buzz about the company, the segment in which the company operates in and the competitive landscape. See if they are covered by reputable national media. A common strategy that some startups use is to make PR releases or pay for favorable coverage. Independent coverage is a good sign that the company is legit and is getting traction. When reading feature articles about the company, look for ranking. Many publications will provide a listing of the top leaders in the domain. If your company is not in the top list and is just being mentioned using language similar to “also active in this space is…”, this could be a sign that they paid the publisher just to get into print.
Team Makeup – In software more so than in most other engineering disciplines, the human factor and the work environment are critical to success. A salt mine culture and a dysfunctional team are indications that the company will perform poorly. When evaluating the team, inquire about the FTE to contractor ratio. Heavy offshore presence could be an indication that the company is a façade with the bulk of the architecture, development, and engineering work being done offsite/offshore by some outsourced firm. This could a problem if you are under the impression that you are investing in domestic IP and human capital.
Work Culture – The work culture is a good indicator of how functional the organization is. Find out if they are burning the midnight oil every day and if so, why? Are they fixing bugs? Trying to catch-up on backlog features? Working long hours in a startup is the norm, but doing it for long periods of time could be an indication that they have not yet found their stride. Ask questions like: “What do you love and hate about the company?” or “If you could change three things, what would they be?”
Compensation – This may not be obvious but compensation can teach you a lot about how well the company is doing. Working in a startup requires some financial tradeoffs but the compensation for the technical team should be within/above the standard industry pay rates. The company should not run like a charity. Did the team get their bonuses last year? Missed yearly bonuses and compensation that is low on cash and high in stock options should raise red flags about how well the company is doing.
Now that you have your finger on the pulse of the organization you are ready to separate the wheat from chaff by identifying the most important takeaways about your target company.
As you complete the two previous DD phases, you will most likely discover that not all of the representations made to you were correct, nor were your original assumptions. The objective of this last exercise is to draw a critical line in the sand that if crossed will result in your walking away from the deal.
The following is my list of eight key assumptions that must pass validation:
1. Platform stability – This covers production matrix such as up-time, downtime, maintenance windows, and singed SLAs. The solution must have published SLA and a historical record of past system shutdowns. All systems go down for one reason or another. It’s important that you understand how frequently their system/sub systems bounce and what the reasons are. The need to babysit the system 24X7 or having a large IT to development ration can be an indication that the solution is on constant life support.
2. Ease of deployability – This covers questions such as hosting (cloud based vs. hosted), provisioning, and the mechanisms for deployment of new customers and users. When it comes to creating new customer environments, look for manual steps used for copying code, configuring/populating databases, and the usage of script to create work regions. Clearly, any manual process for setting up and boarding customers and the need to manipulate the back-end through manualy is a big no-no.
3. Solution scalability – This covers questions regarding number of current transactions per customer, number of customers, daily feed sizes, batch processing schedules, daily feed timeline, and core processing windows. Pay close attention to storage, processing, clustering, and load balancing. Look for obvious signs that the solution will not scale. For example, if the company plans to double its customer base in 12 months, they should already have in place the infrastructure to support such growth. Very few organizations are capable of simultaneously galloping and changing horses mid-stream by making significant alterations to to their storage and load balancing architecture.
4. Maintainability – This covers questions such as production release readiness, customer reporting, and bug tracking. Regardless of how young the company is and their appetite for technology debt, they need to have a functional configuration management, change control, and monitoring capabilities. This doesn’t mean that it’s either HP OpenView or bust. To achieve monitoring, open source tools like Nagios will do. Regardless of the tool, they need to have something in place that is integrated into their solution. Without such controls, they will be flying in the dark, which almost certainly will adversely impact their customers.
5. Disaster recovery, business continuity planning, and availability – This covers questions like how and if the company will recover from various disaster scenarios. What happens if they lose a customer database or the records of important transactions? Is this data being backed up daily? Have they ever attempted to recover from backups? If the company is providing financial services or uses big data, find out how they backup the sensitive information such as PCI data and the terabytes of records on their HDFS.
6. Sophistication of intellectual property – This covers questions regarding the robustness of the algorithms, the structure of the data models, the coupling of the various tiers, the utilization of new and cutting edge frameworks, (i.e. big data components like CPE, queue, plug-ins like R, etc.), and how well everything is mashed together. Remember, just because they use cloud storage/hosting or Hadoop doesn’t mean that their solution can achieve their business objectives or even successfully process large amounts of data.
7. Support for internationalization – This covers questions regarding multi-lingual support, localization, redundant hosting and customer support that follows the sun. Very few startups will be able to fully support internationalization. If you are planning to offer this solution as part of your international portfolio of products, you will need real internationalization that goes beyond the skin deep ability to customize logos and labels. Just like in the case of scalability question, if the functionality is not there now, it will require a significant development effort downstream.
8. Security and privacy – This covers questions regarding authentication, anonymization, encryption, sensitive data storage, data retention, compliance with PCI, FFIEC, etc. Security, due to its nature, is viewed almost universally as overhead and an afterthought. If the platform you are evaluating needs to run silent and deep in hostile waters, you need to make sure that areas such as intrusion detection/prevention, access controls, malware/firewall management, and auditing are up to snuff. Look for up-to-date security policies, records of ongoing security audits (SAS 70, CISA, etc.), vulnerability assessments reviews, and penetration tests. If the company has no such records on file, this can be a strong indication of poor security planning, which is a ticking liability time bomb.
General Consideration During your Due Diligence
My primary indicator of readiness and prospect for success is the number of customers that currently use the software. Obviously these numbers may vary with the type of the solution but if your investment target has a steady and growing customer base, they have at least survived the valley of death and are for real. When evaluating the customer base, look for active accounts that use the system regularly. In many startups, the customers are often made up of relatives/friends and pilot users, although, these types of accounts are important for testing they have little commercial value.
Remember, in the end, it doesn’t matter how compelling the business case may seem, what great technologies they have, or how modular their solution architecture is, without a real customer base, it’s a risky gamble.
A secondary indicator is that of the team and organization. Are you are just buying the software, the team, or the entire package? If you are only interested in the IP, then you will need to identify and secure the architects, lead developers, and core technical team in order to assimilate the technology. On the other hand, if you want the product, then you will need to insure that the organizational structure will be maintained. This is not an easy thing to do, as often many core team member will cash their chips and move on to pursue other opportunities after the sale of the company.
A third indicator is that of Intellectual Property. You need to carefully address IP questions and determine who owns it, where the inventions come from, who was exposed to the inventions, what are the rights of the FTE/contractors to these ideas, and if there are any invention disclosure forms or patent filings in place.
An in-depth evaluation of the architecture through a code review of the key algorithms, data structure, and framework that form the secret sauce should help answer most of these questions. It is important that you conduct this discovery hands-on by reviewing code and metrics such as code quality, code complexity, and unit test coverage. This is the only way for you to insure that the magic is real.
Executing an effective technology due diligence is more of an art than a science because each software solution you will evaluate is unique. Many early and mid stage startups need to trade off between delivering basic business value and developing a fully mature prime time ready platform. These competing factors make it hard to determine with certainty if a solution has the potential evolve into a commercial success or if it is just being held together with chicken wire and chewing gum.
It is important to approach each discovery phase with a set of simple objectives that are critical for a favorable evaluation of the overall solution. This way, during the evaluation of each key assumption, you will be able to clearly identify the main decision gates and confidently make a go/no-go determination.
© Copyright 2014 Yaacov Apelbaum. All Rights Reserved.
Argumentum ad populum is that the best way to beat your competitor is to one-up him. If he has 2 features, you need 3 (4 would be even better). If he is budgeting 1 x $ on his solution, you need to spend 2 x $.
This arms race style upmanship is a dead-end. It’s a prohibitively expensive, reactionary, and is a schizophrenic paranoid way of developing software. Organizations following this philosophy can’t envision the future, they can only dwell in the past. They don’t lead, they follow by imitation. If you want to rapidly build a leading solution and not merely mimic existing products, you need to focus on innovation and quality.
The Project Management iron triangle model proposes that every project is controlled by the following four elements:
- Scope (features and Functionality)
Illustration 1. The PM Iron Triangle
Some project managers believe that you can only pick three out of the four. I don’t agree, because quality should never be compromised. would you buy a new car knowing that it has a defective engine or transmission? Most likely not, regardless of how many features it has or if it shipped on time to the dealer. Quality is ‘given’, and as such, it should always be baked into the scope of the project.
I’ve seen numerous projects who compromised on quality by reducing unit test coverage and then engaging in creative bug severity reclassification, just to catch up with a slipping shipping date. In the end, no amount of Bondo and marketing spin can cover up the quality holes in your delivery. Still not convinced, check out the annals of software development screw-ups , they are full of product humdingers that shipped on time, on budget, and scope, but failed miserably because of substandard quality.
So what do you do when the going gets tough? How do you make that difficult decision and chose the right sacrifice for the greater good? And while at it, what is the greater good anyway?
In this posting I’ll share with you a few insights into selecting the lesser of two evils, and some specific project hostage negotiation techniques that may help you get out of some tight delivery spots.
The Project Web
On the surface, each option from the list above seems to be equally weighted, but in reality, It’s not as simple. In a dynamic living and breathing project, Schedule, Budget, and Scope actually carry different degree of importance and impact each other in subtle ways.
Illustration 2. The Project pyramid
It turns out that rather then a interwoven triangular knot that is made up of equal sides that can be resized and traded equally, the project elements actually resemble an intricate spider web (illustration 3). The structure is controlled by many dynamic factors. As you apply force (i.e. poll or push) to any strand in the web, the whole web flexes and deforms unevenly in multiple axis. Each change you apply to the project (adding scope, increasing the budget, and etc.) will ripple through the entire project structure like a pebble dropped in a pond.
Illustration 3. The Project Element Web
As you can see in Illustration 4, increasing the budget and decreasing the schedule will adversely impact the schedule, scope and quality. This action will also unproportionally deform all project axis and introduce a negative self reinforcing feedback loop. This is because the team will now need to quickly add additional development resources to complete the additional work, this will result investing less time on actual development on more time on knowledge transfer, which will introduce more bugs, which will ultimately result in a reduction of the overall quality of the software.
Illustration 4. The Project Element Web – Impact of Budget Increase and Schedule Decrease
This is counter intuitive to popular wisdom which states that increasing the budget for project can allow you to complete the work faster with more functionality.
The Project Annihilator Shooter
Software product development lifecycle is in many ways similar to a shooter video game. You play each scenes collecting, oxygen, ammunition, and kill points. If you are successful and survive enough missions without losing your vitals (i.e. reputation, stamina, leadership, etc.), you get to move to the next level and play again. if you run out of ammunition, oxygen, or your life gauge falls below some critical level, you’re out of the game.
Following this corollary, your ability to continue to play the software development equivalent of Last Light, depends on your ability to juggle all the elements that go into a project.
the following are your variables:
Ammunition/Oxygen = Funding/Budget
Scene Length = Schedule and drop timeline
Number of Targets per Scene= Number of Features of your project
Here’s an easy way to launch on time and on budget: keep features minimal and fixed. Never throw more time or money at a problem, just scale back the scope. There’s a myth that states that: you can launch on time, on budget, and on scope, while keeping quality in check. Well, it’s nonsense, it never happens, and when you try, quality always takes a hit.
If you can’t fit everything in within the allotted time and budget then don’t expand the time and budget. Instead, scale back the scope. You will always have time to add more content later – later is eternal, now is short-lived. Launching early a product that’s smaller is better than launching a solution that is full of holes just because of some magical date. Leave the magic to Siegfried, Roy and, their tigers. Your customer will measure you on how well your features work, not on how many of them you’ve delivered.
Still not convinced, here are some more benefits of fixing time and budget, and keeping scope flexible:
Keep Scope Real and Relevant
Less is more. Less code, less documentation, less regression testing. Remove all but the essentials. Trust me, most of what you think is essential is actually not. Keep your solution small and agile.
Use rapid iterations to lower the cost of change. And believe me, if you plan to deliver a functional product and are soliciting your users for input, there is going to be a lot of change. Launch frequently, tweak, launch again, constantly improving your solution will help you deliver just what customers need and eliminates anything they don’t.
Breakdown Delivery to Phases and Prioritize Content
This is an important exercise to go through regardless of project constrains. You have to figure out what’s important for your customer and how you can deliver the most critical functionality in multiple phases. Not being tied to a single monolithic delivery will force you to make some tough decisions, but will also eliminate the need to prostrate yourself in front of the war console.
It may sound corny, but having a portion of a working product in the customer’s machine is by far better than having a full poorly functional solution on your build server.
The ability to quickly change is paramount for your success. Having scope, budget, and schedule fixed makes it tough to change. Injecting scope flexibility will allow you to be opportunistic in resolving technology and operational unknowns.
Remember, flexibility is your friend. so de-scope, reduce features, and break your monolithic delivery plan to smaller drops. In the end, there is no escaping the proverbial SDLC grim ripper and the fact that you will have to fold. The sooner you show your cards, the better you’ll be in the long run.
Dump the Ballast
Delivering less content will allow you to reduce all the fluff that represents managing upwards (i.e. lengthy power point presentations, the fictional development progress reports, etc.) and force you to focus on building the real McCoy.
In larger organizations weeks can be spent on planning roadmaps and arguing about the fine details of each feature with the goal of everyone reaching consensus on what would be the ‘best’ solution.
That may be the right approach for a $$$ mainframe product, but if you are developing in internet time, fast quality shipping is the way to go. You can always trust the user to tell you if your scope is sufficient. If it’s not, you can always fix it in the next phase and re-ship it again.
When it comes to scope, there is no judgment better than that of the customer’s – resist the urge to engage in long-winded scheduling and especially stay a way from:
- Senseless hundreds of page of slideware roadmaps
- Schedules that depend on a multi-year project plans
- Pristine roadmaps that predict the future (in terms of market and customer needs)
- Pie-in-the-sky solutions (technology that is immature or non-existing)
- Reliance on never ending status/staff meetings and project reviews
- Dependency on hiring dozens of new resources in order to meet the increase in scope
- Endless version, branching, and integration debates
- To succeed in building competitive software, you don’t need an excessive budget, a large team, or a long development cycle. You just need to simplify your solution and deliver it to the customer as fast as possible with the highest quality.
Or as Steve Jobs said it “That’s been one of my mantras – focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it’s worth it in the end once you get there, you can move mountains” .
© Copyright 2013 Yaacov Apelbaum. All Rights Reserved.