Word Cities Summit 2014 and XRVision

Yaacov Apelbaum - World Cities Summit 2014 Logo

First day of the WCS—against all odds and after two months of grueling development work, an impossible deadline, and 15 hour days—we delivered and deployed the world’s first standalone, wearable face recognition system.

It is my humble hope that this small contribution of ours will help to make our cities, communities, and public gatherings safer from violent crime and terror.

Yaacov Apelbaum - XRVision Face Recognition System

© Copyright 2014 Yaacov Apelbaum All Rights Reserved.

Choosing Between Schedule, Budget, Scope, and Quality

Yaacov Apelbaum-Choosing Between Options

Argumentum ad populum is that the best way to beat your competitor is to one-up him. If he has 2 features, you need 3 (4 would be even better). If he is budgeting 1 x $  on his solution, you need to spend 2 x $.

This arms race style upmanship is a dead-end. It’s a prohibitively expensive, reactionary, and is a schizophrenic paranoid way of developing software. Organizations following this philosophy can’t plan for the future, they can only react to the past. They don’t lead by innovation, they follow by imitation. If you want to rapidly build a leading solution and not merely mimic existing products, you need to focus on innovation and quality.

The traditional Project Management iron triangle model proposes that every project is controlled by these four elements:

  1. Schedule
  2. Budget
  3. Scope (features and Functionality)
  4. Quality

Yaacov Apelbaum - Iron Triangle of Project Management

Illustration 1. The PM Iron Triangle

Some project managers believe that you can only pick three out of the four variables. I don’t agree, because quality should never be compromised. Would you buy a new car knowing that it has a defective engine or transmission? Most likely not, regardless of how many other features it has or if it shipped on time to the dealer. Quality is not a variable and is a ‘given’, and as such, it should always be baked into the the product.

I’ve seen numerous projects who compromised on quality by reducing unit test coverage and then engaging in creative bug severity reclassification, just to catch up with a slipping shipping date. In the end, no amount of Bondo and marketing spin can cover up the quality holes in your delivery.  Still not convinced, check out the annals of software development screw-ups, they are full of product humdingers that shipped on time, on budget, and scope, but failed miserably because of substandard quality.

So what do you do when the going gets tough? How do you make that difficult decision and chose the right sacrifice for the greater good? And while at it, what is the greater good anyway?

In this posting, I’ll share with you a few insights into selecting the lesser of two evils, and some specific project hostage negotiation techniques that may help you get out of some tight delivery spots.

The Project Web
On the surface, each of the four options from the list above seems to be equally weighted, but in reality, It’s not as simple. In a dynamic living and breathing software project, the schedule, budget, and scope actually carry different degree of importance and impact each other in subtle ways.

Yaacov Apelbaum-Choosing Between Options Pyramid

Illustration 2. The Project pyramid

It turns out that rather than a interwoven triangular knot that is made up of equal sides that can be resized and traded equally, the project elements actually resemble an intricate spider web (illustration 3). This structure in turn is controlled by many variant and dynamic factors. As you apply force (i.e. poll or push) to any strand in this web, the whole web flexes and deforms unevenly in multiple axis.  Each change you apply to the project (adding scope, increasing the budget, etc.) will ripple through the entire project structure like a pebble dropped in a pond.

Yaacov Apelbaum-Project Option Grid

Illustration 3. The Project Element Web

As you can see in Illustration 4, increasing the budget and decreasing the schedule will adversely impact the schedule, scope and quality. This action will also disproportionally deform all project axis and introduce a negative self-reinforcing feedback loop. This is because the team will now need to quickly add more development resources to complete the additional work, this will result investing less time on actual development on more time on knowledge transfer, which will introduce more bugs, which will ultimately result in a reduction of the overall quality of the software.

Yaacov Apelbaum-Project Options Budget Increase Schedule Decrease

Illustration 4. The Project Element Web – Impact of Budget Increase and Schedule Decrease

This is counter-intuitive to popular wisdom which states that increasing the budget for the project can allow you to complete the work faster with more functionality.

The Project Annihilator Shooter
Software product development lifecycle is in many ways similar to a shooter video game. You play each scenes collecting, oxygen, ammunition, and kill points. If you are successful and survive enough missions without losing your vitals (i.e. reputation, stamina, leadership, etc.), you get to move to the next level and play again. If you run out of ammunition, oxygen, or your life gauge falls below some critical level, you’re out of the game.

The Project Annihilator Dashboard

Following this corollary, your ability to continue to play the software development equivalent of Metro: Last Light, depends on your ability to juggle all the elements that go into your software project.

The following are your variables:

Ammunition/Oxygen = Funding/Budget
Scene Length = Schedule and drop timeline
Number of Targets per Scene= Number of Features of your project

Here’s an easy way to launch on time and on budget: keep features minimal and fixed. Never throw more time or money at a problem, just scale back the scope. There’s a myth that states that: you can launch on time, on budget, and on scope, while keeping quality in check. Well, it’s nonsense, it never happens, and when you try, quality always takes a hit.

If you can’t fit everything in within the allotted time and budget then don’t expand the time and budget. Instead, scale back the scope. You will always have time to add more content later–later is eternal, now is short-lived. Launching early a product that’s smaller is better than launching a solution that is full of bugs just because of some magical date. Leave the magic to Siegfried, Roy and, their tigers. Your customer will judge you on how well your features work, not on how many of them you’ve delivered.

Still not convinced, here are some more benefits of fixing time and budget, and keeping scope flexible:

Keep Scope Real and Relevant
Less is more. Less code, less documentation, less regression testing. Remove all but the essentials. Trust me, most of what you think is essential is actually not. Keep your solution small and agile.

Use rapid iterations to lower the cost of change. And believe me, if you plan to deliver a functional product and are soliciting your users for input, there is going to be a lot of change. Launch frequently, tweak, launch again, constantly improving your solution will help you deliver just what customers need and eliminates anything they don’t.

Breakdown Delivery to Phases and Prioritize Content 
This is an important exercise to go through regardless of project constrains. You have to figure out what’s important for your customer and how you can deliver the most critical functionality in multiple phases. Not being tied to a single monolithic delivery will force you to make some tough decisions, but will also eliminate the need to prostrate yourself in front of the war console.

It may sound corny, but having a portion of a working product in the customer’s machine is by far better than having a full poorly functional solution on your build server.

Be Flexible
The ability to quickly change is paramount for your success. Having the scope, budget, and schedule fixed makes it tough to change. Injecting scope flexibility will allow you to be opportunistic in resolving technology and operational unknowns.

Remember, flexibility is your friend. so de-scope, reduce features, and break your monolithic delivery plan to smaller drops. In the end, there is no escaping the proverbial SDLC grim ripper and the fact that you will have to fold. The sooner you show your cards, the better you’ll be in the long run.

Dump the Ballast
Delivering less content will allow you to reduce all the fluff that represents managing upwards (i.e. lengthy Power Point presentations, the fictional development progress reports, etc.) and force you to focus on building the real McCoy.

In larger organizations, weeks can be spent on planning roadmaps and arguing about the fine details of each feature with the goal of everyone reaching consensus on what would be the ‘best’  solution.

That may be the right approach for a $$$ mainframe product, but if you are developing in internet time, fast quality shipping is the way to go. You can always trust the user to tell you if your scope is sufficient. If it’s not, you can always fix it in the next phase and re-ship it again.

When it comes to scope, there is no judgment better than that of the customer’s–resist the urge to engage in long-winded scheduling and planning and especially stay away from:

  • Senseless hundreds of page of slide ware roadmaps
  • Schedules that depend on a multi–year project plans
  • Pristine roadmaps that predict the future (in terms of market and customer needs)
  • Pie-in-the-sky solutions (technology that is immature or non-existing)
  • Reliance on never-ending status/staff meetings and project reviews
  • Dependency on hiring dozens of new resources in order to meet the current increase in scope
  • Endless version, branching, and integration debates

To succeed in building competitive software, you don’t need an excessive budget, a large team, or a long development cycle. You just need to simplify your solution and deliver it to the customer as fast as possible with the highest quality.

Or as Steve Jobs said it “That’s been one of my mantras – focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it’s worth it in the end once you get there, you can move mountains”.

© Copyright 2013 Yaacov Apelbaum. All Rights Reserved.

The API Train Wreck Part-2

ApelbaumYaacov API Train Wrack 2

In The API Train Wreck Part-1, I discussed API design factors such as KPI, performance measurements, monitoring, runtime stats, and usage counters.  In this posting, I’ll discuss the design factors that will influence your API’s load and traffic management architecture.

Load and Traffic Management
One of the key issues that you are going to face when opening your API to the world are the adverse effects on your own front-end and back-end systems.  If you are using a mixed model, where the same API is used for both internal line of business applications and external applications, uncontrolled bursts of traffic and spikes in data requests can cause your service to run red hot.

In a mixed model, as the number of external API users increases, it’s only a question of time before you will start experiencing the gremlin effects which include brownouts, timeouts, and periodic back-end shutdowns. Due to the stateless nature of SOA and the multiple points of possible failure, troubleshooting the root cause of these problems can be a nightmare.

This is why traffic management is one of the first and most important capabilities you should build into your API. A good example of this type of implementation can be found in the Tweeter REST API Rate Limiting. Twitter provides a public API feed of their stream for secondary processing. In fact, there is an entire industry out there that consumes, mines, enriches, and resells Twitter data, but this feed is separate from their internal API and it contains numerous traffic management features such as: caching, prioritize active users, adapting to the search results, and blacklisting to insure that their private feed will not be adversely impacted by the public API.

Bandwidth Management
Two of the most common ways to achieve efficient bandwidth management and regulate the flow are via (1) throttling and (2) rate-limiting.  When you are considering either option, make the following assumptions:

  • Your API users will write inefficient code and only superficially read your documentation. You need to handle the complexity and efficiency issues internally as part of your design.
  • Users will submit complex and maximum query ranges.  You can’t tell the user not to be greedy, rather you need to handle these eventualities in your design.
  • Users will often try to download all the data in your system multiple times a day via a scheduler. They may have legitimate reason for doing so, so you can’t just deny their requests because it’s inconvenient for you. You need to find creative methods to deal with this through solutions like of-line batch processing or subscription to updates. 
  • Users will stress test your system in order to verify its performance and stability. You can’t just treat them as malicious attackers and simply cut off their access. Take this type of access into account and accommodate it through facilities like a separate test and certification regions.

In order to be able to handle these challenges, you need to build a safety fuse into your circuitry.  Think of Transaction Throttling and Rate Limiting as a breaker switch that will protect your API service and backend.

Another reason to implement transaction throttling is because you may need to measure data consumption against a time based quota.  Once such mechanism is in place, you can relatively easily segment your customers by various patterns of consumption (i.e. data and time, volume, frequency, etc.).  This is even more relevant if you are building a tier based service where you charge premium prices per volume, query, and speed of response.

Rationing, Rate Limitations, and Quotas
OK, so now we are ready to implement some quota mechanisms that will provide rate limits. But just measuring data consumption against daily quotas can still leave you vulnerable to short bursts of high volume traffic or if done via a ‘per second ‘ rate limits can be perceived as a non-business friendly.

If your customers are running a high volume SaaS solution and are consuming your API as part of their data supply chain, they will probably find it objectionable when they discover that you are effectively introducing pre-defined bottlenecks into their processing requests.

So even if you consider all data requests equal – you may find that some requests are more equal than others (due to the importance of the calling client) or that they contain high rate transactions or large messages, so just limiting ‘X requests per second’ might not be sufficient.

Here are my core 13 considerations for designing the throttling architecture.

  1. Will you constrain rate of a particular client access by API key or IP address?
  2. Will you limit your data flow rate by user, application, or customer?
  3. Will you control the size of messages or the number of records per message?
  4. Will you throttle differently your own internal apps vs. public API traffic?
  5. Will you support buffer or queue based requests?
  6. Will you offer business KPIs on API usage (for billing, SLA, etc.)?
  7. Will you keep track of daily or monthly usage data to measure performance?
  8. Will you define different consumption levels for different service tiers?
  9. How will you handle over quota conditions (i.e. deny request, double bill, etc.)?
  10.   Will you measure data flow for billing and pricing?
  11.   Will you monitor and respond to traffic issues (security, abuse, etc.)?
  12.   Will you support dynamic actions (i.e. drop or ignore request, send an alert, etc.)?
  13.   Will you provide users usage metadata so they can help by metering themselves?

Obviously, no single consideration in the list above will singlehandedly control your design. During your deliberation process, dedicate some time to go over each of the 13 points. List the pros and cons for each and try to estimate the overall impact on the development scope and overall project timeline.

For example, evaluating  “1. Constrain rate of a particular client access by API key” will yield the following decision table:

Pros for Key Management

Cons for Key Management


Conducive to per-seat and volume licensing models and can support both enterprise and individual developers.

Need to invest in the setup and development of a key lifecycle management system.

Effort is estimated at x man months and will push the project delivery date by y months.

Tighter control over user access and activity.

Need to provide customer support to handle key lifecycle management support.

Effort will require to hire/allocate x number dedicated operational resources to support provisioning, audit, and reporting.

Conducive to a tiered licensing and billing model including public, government, and evaluate before you buy promotions.

Managing international users will require specialty provisioning and multi-lingual capabilities.

Will complement and support the company’s market and revenue strategy.

Will scale well to millions of users.

Initial estimate of number of customers is relatively small.


Resist the temptation to go at it ‘quick and dirty’. Collect as much input as possible from your front-end, middle tier, back-end, operations, and business stakeholders. Developing robust and effective throttling takes careful planning and if not done correctly can become the Achilles heel of your entire service.

The above pointers should cover the most critical aspects of planning for robust API traffic management.

© Copyright 2013 Yaacov Apelbaum. All Rights Reserved.

The API Train Wreck Part-1

ApelbaumYaacov API Train Wrack

Developing a commercial grade API wrapper around your application is not a trivial undertaking. In fact, in terms of effort, the development of an outbound facing API can dwarf all combined application development tasks by an order of magnitude.

I’ve seen some large development organizations attempt to open their internal platform to the world just to watch in horror as their core systems came to a screeching halt before ever reaching production.

In this and a follow-up posting, I’ll try to provide you with some hands-on tips about how you can improve the quality of your API and more efficiently ‘productize’, ‘commercialize’, and ‘operationalize’ your data service.

It’s 9 PM, do you know who’s using your API?
Do you have accurate and up-to-date KPIs for your API traffic and usage? Are these counters easily accessible or do you need to troll through numerous web server and application logs to obtain this data? Believe it or not, most APIs start as orphaned PoCs that were developed with the approach of ‘do first, ask permission later’.  Things like usage metrics are often put on the back burner until project funding and sponsorship roll in.

Paradoxically, due to their integrative effect, application APIs can provide the biggest boost to traffic and revenue, but at the same time they can become your biggest bottleneck.

So my first tip is to build usage information into your core functions. Before you embark on an API writing adventure, your product management and development teams should have clear answers to these questions:

How many clients, applications, and individual developers will use your API?

  • Will you offer public and private versions of it?
  • What will the distribution of your clients, apps, and developers by type and location be?
  • Who will the the top tier customers be? Developers? Partners?
  • What parts of the API will be used most heavily?
  • What will the traffic breakdown between your own services and 3rd party services be?
  • What will the aggregate per application and per customer transaction volumes be?
  • How fast and reliable will our service be, (i.e., response, latency, downtime, etc.)?
  • What are the architectural and technical limiting factors of your design (i.e. volume)?
  • How will the API be monitored and SLAd?
  • What type of real-time performance dashboards will you have?
  • How will the public facing API usage effect the rest of your platform, (i.e., throttling)?
  • Will you handle error routines generically or identify and debug specific messages?
  • Do you need to support auditing capabilities for compliance, (i.e SOX, OFEC, etc.)?
  • What will the volume of the processed/delivered data be, (i.e. from a third parties)?

Asking these questions—and getting good answers to them—is critical if you plan to develop a brand new SaaS platform to expose your internal platforms to the world.

This information is also critical to your ability to establish contractual relationships with your internal users and external customers. Depending on the type of information you serve, it may be also mandatory for compliance with state and Federal regulations.

Collecting the correct performance metrics and mapping them to your users and business functions will help you stay one step ahead of your user’s growing demands.

© Copyright 2013 Yaacov Apelbaum. All Rights Reserved.

To ESB or not to ESB is the question. Or is it more noble to just ETL it?

ESB vs. ETL-Yaacov Apelbaum

To borrow a metaphor from Winston Churchill, for the mere mortal, the selection of application/platform/enterprise integration tools, is often “a riddle wrapped in a mystery inside an enigma”. If your conversation with your enterprise architect left you disoriented and you feel like the right choice of enterprise integration tools requires mastery of ancient Sumerian incantations, then you’ve come to the right place.  Sit back and relax. In this posting, I’ll try to demystify the black art of choosing between ETL and ESB middleware technologies, discuss their operational requisites, and touch on the advantages of each. 

ESB and ETL under the Microscope
ETL (Extract-Transform-Load) and ESB (Enterprise Service Bus) share some conceptual and operational similarities but are used structurally for different purposes. ESB  is used typically as the mediation layer in SOA (Service Oriented Architecture) implementations, whereas ETL is used as a data transfer/transformation engine from one database/data depository to another.

A good analogy for how/what type of service bus to choose, (ESB, ETL, messaging, queue, etc.), is the motherboard architecture in PCs and laptops. Your ability to add new services to your platform, (i.e. a new data source), is similar to the strategy you would use to plug future hardware into your computer. Two common methods are PCI and USB.

Because PCI sits on the main bus and operates close to the on-board controllers and processor, it can replace existing hardware, (like the default video card, or even the service bus itself). PCI gives you tremendous performance advantages, but to use it you have to master many OS and hardware nuances and deal with some low level plumbing. On the other hand, if you have a laptop, the motherboard architecture doesn’t support PCI, (that’s what makes the laptop portable), so the most common way to expand it is via USB. Due to its ‘universal’ nature, USB doesn’t provide the same low level system integration as PCI, but it does solve most common peripheral integration needs.

To the ends user, the end result of the PCI and USB architectures is almost identical. Both allow him to add new hardware to his computer, but each has implementation and mission objectives which are significantly different. The same applies to ESB, (products like OSB or WebSphere), and business process integration tools, (products like Talend and Ab-Initio). Whereas an ESB is measured by its deep coupling, ETL/BPI tools are measured by their ability to add new data sources and create faster data mappings and transformations.

Viva la petite difference
The differences between ESB and ETL are important when choosing an enterprise integration strategy. If your platform is SOA ready, it has well defined interfaces and you know what the operational and physical boundaries of your solution are, (inside a firewall, on same network, etc.). A full-fledged ESB, (the PCI equivalent), is the most efficient way to go, because it will give you high volume data flow, synchronous messaging, orchestration, and reliable pub/sub capability. But if you are just looking to solve simple integration issues like connecting several legacy applications to your backend, an ESB is overkill and a solution like ETL, (the USB equivalent), would do.

Use Cases
A good use case for how ESBs should be used is where you have various web services that you want to expose internally and externally. You can host these web services on a ESB as an external interface to allow other Systems/Service to access them. If for some reason, the interface of the Web Services changes, you can handle it via the ESB itself by using transformation, so that external services/systems don’t have to change the service interface. Your ESB will still be able to receive data required by the previous interface, transform it and call a new interface so that no other system/service will be impacted by the web interface changes.

ETL Selection Process
ETL has a relatively simple use case selection criteria. If you all you want to do is achieve the following functions, this should be your tool of choice:

  • Move and transform data across multiple data sources
  • Normalize data (you can also achieve this via an ESB but it won’t be as powerful as via an ETL unless your write some heavy duty wrapper code)
  • Improve or replace your batch processing engines
  • Improve efficiency and scale of your mass data loading operations.

ESB Selection Process
There are no hard and fast rules for rolling out an ESB, but it helps if your systems/platforms already have the following capabilities:

  • Most of your components speak or understand XML
  • You have abstracted interfaces
  • The platform components have basic SOA capabilities, (services adhere to agreements, are loosely coupled, hide logic from the outside world, are stateless, and are discoverable)
  • Majority of manual and hard wired business processes like batch jobs have been wrapped in by web services, (preferably using REST)
  • You need to orchestrate communications between internal applications, (i.e. ERP to CRM) or to build composite applications, (web portal presenting data from various applications)
  • Integrate information, (not only data), sources from heterogeneous environment, (intranet, extranet, line-of business applications)
  • Operate on real-time data and service many messages with small data sizes, (i.e. single bank transaction, single order information, etc.)

Now, assuming that you do have many of the above mentioned capabilities and you know that in the future you will need to achieve some of the following objectives, then an ESB is a good fit for your needs:

  • Integrate many more applications (from inside and outside of your enterprise)
  • Plug more services into our platform, (your solution will constantly evolve)
  • Need more core capabilities (communication protocols, real-time support, transactional capabilities, etc.)
  • Handle different types of messages (content based routing, message forking, aggregation, etc.)
  • Work with SLA based services, (provide audit, tracking, reconciliation, etc.)
  • Leverage/coexist with unknown technology stacks and architectures
  • Develop a data supply chain or business solutions that require complex orchestrations

A Final Note
One of the biggest challenges of forcing ESB into an existing environment is that ESB functions optimally on an OSA based architecture, but it isn’t a prerequisite for it. Due to the fact that most ESBs also provide integration tools, they can encourage architects to cut corners, and instead of taking the time to tear down application silos and rebuild them according to SOA guidelines, they just use the ESB to patch-up these silos and continue to operate them indefinitely. Obviously, this can becomes an attractive proposition when a project is late and under pressure to integrate fast.

© Copyright 2013 Yaacov Apelbaum, All Rights Reserved.

Your First 90 Days on the Job

Yaacov Apelbaum-First 90 Days

You just landed the technology executive job of your dreams, so what do you do next?  Do you concentrate on the dysfunctional engineering/architectural environment? Do you accelerate the delivery schedule on the flagship revenue platform? Do you tackle that daunting platform upgrade that everybody has been steering clear of for the past several years? Well, you should do none of those, and I’ll tell you why.

Your first ninety days on the job are critical to your long term success. This transition period is characterized by great uncertainty and the risk of deadlocks.  As many eventually discover, the inability to make significant headway during this initial period has little to do with technical intelligence or leadership skill. The common cause for the ninety day blues has to do with communication breakdowns, misaligned expectations, and the onset of a psychiatric condition known as the “Hero Syndrome”.

If you have just given notice and are about to take over new technical management responsibility, you may want to consider the following tips and tricks:

1) Committing to unrealistic deliverables and timelines
The number one trap on my list is to promise too much and commit to specific deliverables too early. Creating unrealistic expectations is one of the most common pitfalls for a new manager. It is all too human to endeavor to dazzle your boss and peers by establishing the reputation of a mover and shaker. Fight this urge until you get a good understanding of what it is exactly that you are about to move and shake.

One great example to illustrate this is the story taken from the Mahabharata  about king Santanu, who commits to a deadly relationship with Ganga without first understanding the repercussions of his decision. It cost him 7 of his 8 children.

2) Pretending you know it all
The second pitfall is believing that you have been there and done that. Regardless of your pedigree and illustrious career, no two projects are ever the same.  You must come to terms with the fact that you can’t possibly have all the answers. My favorite tactic for addressing this is to schedule a day long summit that is dedicated to one core problem.  During the summit, the stake holders present and openly discuss their concerns and views. Spend the time listening to all the domain experts. Don’t be shy, if you don’t understand the issues discussed, ask for a simplified explanation.  In all likelihood, when the fog eventually lifts, you will find that the problem is more complex than you had initially thought.

Another variation on the knowing-it-all syndrome is jumping to conclusions. Quickly embracing a substandard solution as a simple fix to a complex problem can alienate your organization. Team members who believe their leaders’ minds are made up about a problem are usually reluctant to share information or ideas, and this can further impede your ability to learn the true nature of the issue.

3) The good ol’ days and clinging to your old persona
Sometimes without even realizing it, you may discuss your former company or your past success too frequently. The psychology behind this behavior is to use past achievements (real or imaginary) to shore up an argument. This may work some of the time, but it is a double edged sword because doing it habitually can disenfranchise your new team and create the impression that your former company was so much better than your current one, at least in your mind. People may also wonder why—if you loved your old company so much—you didn’t marry it and stay there.

4) Suppressing bad news and dissent
Managers who quash disagreements, bad news, and dissent remove themselves from the real feedback loop and lose the ability to identify and correct problems. It’s easy to create a top to bottom environment that is based on fear. This, however, is guaranteed to drive your brightest team members out the door. The highways are full of smart employees who were given draconian decrees in the style of “it’s my way or the highway.” The only ones left in your company parking lot will likely be the mediocre talent who have difficulty driving to their next gig.

5) Chronic Hero Syndrome
Trying to go at it solo is romantic but foolish. If you operate as a lone wolf who insists on forging his/her own way and doing all the work yourself (design, coding, architecture, etc.), you will cut yourself off from valuable sources of domain expertise. Even if you are on the right track, you will eventually burn out or alienate your team.

6) Failing to identify the true sources of corporate power
One important quality you should work on is reading the unwritten laws of your organization.  Depending on the structure of your new company, (startup v. Fortune 500), you may find that real power and ability to effect change is held by the likes of senior architects, product managers, etc. and not your C-class peers.

7) Fighting the wrong battles
As a new leader, you instinctively want to focus on high visibility problem areas and figure out how to solve them. Nothing is more rewarding than to take a leading role in a gunfight at the O.K. Corral. That’s commendable, but not if it comes at the expense of sustaining small gains at GTB (growing the business) while suffering significant losses at RTB (running the business). There is a tendency in the beginning to think that it’s more important to be visible. Fighting too many of these battles can quickly become a huge  black hole that will suck every free moment of your day.

8) “Trashing” your predecessor
My golden rule is to always be respectful and sensitive about my predecessors on the job.  Consider the fact that most of the people you will work with knew the old manager and many are still in touch with him/her and are probably still on friendly terms. Learn as much as you can about your predecessor and the reason for his/her actions. Try to figure out what was done well and what went wrong. If at all possible, insist on getting access to old e-mails and work files. As I have discovered many times, this data can contain pure gold and is probably a good indication of the type of bus that is heading your way.

9) Attempting too much
Whether you like it or not, you are going to be judged on specific deliverables. If you concentrate your focus on too many activities, you may miss the opportunity to allocate a critical mass resources and time to complete high priority projects. To avoid this pitfall, work to prioritize your assignments. My rule is no more than three vital priorities for the first 90 days.

10) Hanging around with the wrong crowd
In life, hanging around with the wrong crowd can get you into trouble. The same thing applies to your new team. If you get inadequate or skewed information, or make all your decisions based on poor advice that comes only from a handful of individuals on your team, you may end up inadvertently politicizing trivial issues. Keep the lines of communication open outside of your organization in order to make sure that you balance inter-team influences.

11) Failing to build a cross-organizational coalition
It’s impossible to get anything done effectively in any size organization without critical mass of support. This is even more important for large corporate initiatives. For you, this means that you very quickly need to identify who the key decision makers are and what they care about, map the organization’s political networks to figure out who influences the influencers, and craft a plan to reach out to build strategic relationships.

Your Recipe for Success
Avoiding the above mentioned traps depends on how well you manage your first ninety days on the job. This means taking the time to evaluate situations, identifying team member strengths/weakness, accelerating learning, negotiating success, building coalitions, and achieving early victories.

Coming into a senior management position from the outside of the organization is tough. You don’t have a reputation to fall back on yet nor an internal Goombah to speak up for you. You must quickly learn about new products, market opportunities, and the mechanics of day-to-day operations – and all this in the context of an unfamiliar political culture.

Yaacov Apelbaum - Werewolf Killing KitTo execute your vision successfully, avoid using political clout and authority. Instead,  leverage your team members to get the job done. I use the “clip with 3 silver bullets” metaphor. You can use management directives 3 times, but each time you do, you spend one of your silver bullets. After your last management directive, you are effectively out of ammunition and it’s only a question of time before the werewolves will get you.

This means building a good relationship with team members from other business units who have their own agendas. It means creating supportive relationships with your manager and peers who have different styles and objectives. It means motivating a wide range of employees who probably feel threatened by you. So how do you put this hundred ton flywheel in motion and secure support? You do it by following these battle proven directives:

1) Manage your management team
What makes companies like Intel, Google, and Apple successful is their talented and high-performing employees. Creativity, product innovation, and development efficiency are largely based on good technology management practices.  Whether you are coming in as CTO to resuscitate a fallen giant, or as a VP of engineering in a startup to get their product to market, the primary lesson is that you can’t do it alone.  You need to build a dependable management team that should be able to survive the loss of any of its key members (including you) and still continue to function.  For you, that means working out individual secession plans and career paths.

Neff and Citrin, in their book, You’re in Charge-Now What? argue that the difference between short term success and great enduring success is how you practice leadership and shape your management team in the early days, and then motivate and develop them over a longer period of time.

2) Develop and articulate your strategic plan
In the fog of battle, it may seem as if there is no time to think, but that’s not the case. Take your time and work out your plans. As a new executive, you are not expected to produce a fully functional strategic plan in the first ninety days. You should instead try to strike a balance between developing a comprehensive map of where you want to take the organization without immediately becoming prematurely bound to it.

The first ninety days are your honeymoon period and as a new manager, you are allowed to walk a fine line. You must develop credibility through action but at the same time,  if you act too quickly before gaining a complete understanding of the situation, you may risk making the wrong choices.

3) Communicate
Communication is probably the most important aspect of good leadership. Effective communication has a disproportionate effect on your success during the early period.

Don’t miss the opportunity to communicate daily and to communicate well.  Listen carefully during meetings, take careful notes (write down and learn every acronym), follow-up on conversations to get in-depth explanations for issues that you didn’t grasp initially, be punctual showing up to meetings and returning e-mails and phone calls, and provide constructive non-polemic feedback.

4) Transform the culture, but do it slowly…
Every organization has a different ecosystem and operational nuances. Before you start transforming the organization, learn about the culture of the company and identify its bureaucratic and operational bottlenecks. It is critical to first learn “how they do things around here” and to identify the knowledge network, secret handshakes, chief influencers, decision-protocols, and unwritten conventions that form the nervous system of your new organization.

A few good areas for investigation are procurement, recruitment, release management, and quality tracking. My two personal favorites are to go through the actual steps of composing and submitting a request for a piece of software or hardware and a new headcount justification request. You will be amazed how much you can learn about the company by completing these two tasks. Once you get your finger on the pulse, you can begin to intelligently effect change and take the right transformational steps. This would include: instituting new operating procedures, choosing a new management team, setting up governance boards, identifying change leaders, and leading by example.

Finally, change is unavoidable, but it is important to remember that rapid and excessive change can destroy the work culture and the change agent (i.e., you). So, instead of adapting a deeply disruptive strategy such as reinventing everything or replacing key people, try the following long terms initiatives:

  • Set up pilot and POC projects
  • Change the way performance is measured and tracked (digital dashboards)
  • Educate and train your team (gain expertise in your entire technology stack)
  • Build up islands of excellence (tiger team, high performance SCRUM teams, etc.)
  • Document how the technology organization operates (process and roadmaps)
  • Collectively envision new ways to operate (what to sunset, upgrade, etc.)

5) Get to know your boss, your greatest ally
A key success factor in your new job is to establish a productive working relationship with your new boss. It is important that you spend some time together on daily basis. This includes keeping him/her informed of your thoughts and actions, discussing proposed changes, and updating him/her on your deliverables.

It is also imperative that you quickly size up your boss, including high priority goals, pain points, strengths, weaknesses, blind spots, and preferred work style.

Remember, your boss is your greatest ally, because he/she can link you to all the right individuals in the rest of the organization, help you set priorities, and secure the resources you need to get your job done.  All of this is free and comes without you having to spend any political capital at all.

For a more detailed treatment on how to max the first 90 days on the job, check out: The First 90 Days: Critical Success Strategies for New Leaders at All Levels by Michael Watkins


© Copyright 2011 Yaacov Apelbaum All Rights Reserved.

Big O Notation

Yaacov Apelbaum-big-o-and-efficiency

Recently, I was chatting with a friend of mine about pre-acquisition due diligence. Charlie O’Rourke is one of the most seasoned technical executives I know. He’s been doing hardcore technology for over 30 years and is one of the pivotal brains behind WU/FDC’s multi-billion dollar payment processing platforms. The conversation revolved around a method he uses for identifying processing bottlenecks.
His thesis statement was that in a world where you need to spend as little as you can on an acquisition and still turn profit quickly, problems of poor algorithmic implementations are “a good thing to have”, because they are relatively easy to identify and fix.  This is true, assuming that you have his grasp of large volume transactional systems and you are handy with complex algorithms.

In today’s environment of rapid system assembly via the mashing of frameworks and off-the shelf functionality like CRM or ERP, the mastery of data structures by younger developers is almost unheard of.

It’s true, most developers will probably never write an algorithm from scratch. But sooner or later, every coder will have to either implement or maintain a routine that has some algorithmic functionality. Unfortunately, when it comes to efficiency, you can’t afford to make uninformed decisions, as even the smallest error in choosing an algorithm can send your application screaming in agony to Valhalla.

So if you have been suffering from recursive algorithmic nightmares, or have never fully understood the concept of algorithmic efficiency, (or plan to interview for a position on my team), here is a short and concise primer on the subject.

First let’s start with definitions.

Best or Bust:
An important principal to remember when selecting algorithms is that there is no such thing as the “best algorithm” for all problems. Efficiency will vary with data set size and availability of computational resources (memory and processor). What is trivial in terms of processing power for the NSA, could be prohibitive for the average company.

Algorithmic efficiency is the measure of how well a routine can perform a computational task. One analogy for algorithmic efficiency and its dependence on hardware (memory capacity and processor speed) is the task of moving a ton of bricks from point A to point B a mile a way.  If you use a Lamborghini for this job (small storage but fast acceleration), you will be able to move a small amount of bricks very quickly, but the down side is that you will have to repeat the trip multiple times. On the other hand, if you use a flatbed truck (large storage but slow acceleration) you will be able to complete the entire project in a single run, albeit at slower pace.

The expression for algorithmic efficiency is commonly referred to as “Big O” notation.  This is a mathematical representation of how the algorithm grows over time. When plotted as a function, algorithms will remain flat, grow steadily over time, or follow varying curves.

The Pessimistic Nature of Algorithms:
In the world of algorithm analysis, we always assume the worst case scenario.  For example, if you have an unsorted list of unique numbers and it’s going to take your routine an hour to go through it, then it is possible in the best case scenario that you could find your value on the first try (taking only a minute). But following the worst case scenario theory, your number could end up being the last one in the list (taking you the full 60 minutes to find it). When we look at efficiency, it’s necessary to assume the worst case scenario.

 Yaacov Apelbaum-big-o Plot
Image 1: Sample Performance Plots of Various Algorithms

Performance is constant for time (processor utilization) or space (memory utilization) regardless of the size of the data set size. When viewed on a graph, these functions show no-growth curve and remain flat.

O(1) algorithm’s performance is also independent of the size of the data set on which it operates.

An example of this algorithm is testing a value of a variable based on some pre defined hash table.  The single lookup involved in this operation eliminates any growth curves.

Performance will grow linearly and in direct proportion to the size of the input data set.  The algorithm’s performance is directly related to the size of the data set processed. 

O(2N) or O(10 + 5N) denote that some specific business logic has been blended with the implementation (which should be avoided if possible).

O(N+M) is another way of saying that two data sets are involved, and that their combined size determines performance.

An example of this algorithm is finding an item in an unsorted list or a Linear Search that goes down a list, one item at a time, without jumping.  The time taken to search the list gets bigger at the same rate as the list does.

Performance will be directly proportional to the square of the size of the input data set.  This happens when the algorithm processes each element of a set and that processing requires another pass through the set (this is the square value). Processing a lot of inner loops will also result in the form O(N3), O(N4), O(Nn.).

Examples of this type of algorithm are Bubble Sort, Shell Sort, Quicksort, Selection Sort or Insertion Sort.

Processing growth (data set size and time) will double with each additional element of the input data set. The execution time of O(2N) can grow exponentially.

The 2 indicates that time or memory doubles for each new element in data set.  In reality, these types of algorithms do not scale well unless you have a lot of fancy hardware.

O(log n) and O(n log n) 
Processing is iterative and growth curves peak at the beginning of the execution and then slowly tapper off as the size of the data sets increases.  For example, if a data set contains 10 items, it will take one second to complete; if the data set contains 100 items, it will takes two seconds; if the data set containing 1000 items, it will take three seconds, and so on. Doubling the size of the input data set has little effect on its growth because after each iteration the data set will be halved. This makes O(log n) algorithms very efficient when dealing with large data sets.

Generally, log N implies log2N, which refers to the number of times you can partition a data set in half, then partition the halves, and so on.  For example, for a data set with 1024 elements, you would perform 10 lookups (log21024 = 10) before either finding your value or running out of data.

Lookup # Initial Dataset New Dataset
1 1024 512
2 512 256
3 256 128
4 128 64
5 64 32
6 32 16
7 16 8
8 8 4
9 4 2
10 2 1

A good illustration of this principal can be found in the Binary Search, it works by selecting the middle element of the data set and comparing it against the desired value to see if it matches. If the target value is higher than the value of the selected element, it will select the upper half of the data set and perform the comparison again. If the target value is lower than the value of the selected element, it will perform the operation against the lower half of the data set. The algorithm will continue to halve the data set with each search iteration until it finds the desired value or until it exhausts the data set.

The important thing to note about log2N type algorithms is that they grow slowly. Doubling N has a minor effect on its performance and the logarithmic curves flatten out smoothly.

An example of these type of algorithms are Binary Search, Heap sort, Quicksort, or Merge Sort

Scalability and Efficiency
An O(1) algorithm scales better than an O(log N),
which scales better than an O(N),
which scales better than an O(N log N),
which scales better than an O(N2),
which scales better than an O(2N).

Scalability does not equal efficiency. A well-coded, O(N2) algorithm can outperform a poorly-coded O(N log N) algorithm, but this is only true for certain data set sizes and processing time. At one point, the performance curves of the two algorithms will cross and their efficiency will reverse.

What to Watch for when Choosing an Algorithm
The most common mistake when choosing an algorithm is the belief that an algorithm that was used successfully on a small data set will scale effectively to large data sets (factor 10x, 100x, etc.).

For most given situations, an O(N2) algorithm like Bubble Sort will work well. If you switch to a more complex O(N log N) algorithm like Quicksort you are likely to spend a long time refactoring your code and will only realize marginal performance gains.

More Resources
For a great illustration of various sorting algorithms in live action form, check out David R. Martin’s animated demo.  For more informal coverage of algorithms, check out Donald Knuth’s epic publication on the subject The Art of Computer Programming, Volumes 1-4.

If you are looking for some entertainment while learning the subject, check out AlgoRythimic’s series on sorting through dancing.


© Copyright 2011 Yaacov Apelbaum All Rights Reserved.