A recurring challenge in software — especially for newly developed systems or major upgrades — is knowing when you’re ready to deploy or ship for the first time in production. When is “good enough” actually “good enough?”

In this article, we’ll explore this concept of good enough and the nuance around it. You’ll learn key differences between internal and external success, how ship decisions are made by various stakeholders, and where teams and individuals often disagree on precisely what makes a product or update “good enough” and ready to go. Finally, we’ll work through different approaches using real-world use cases and see what lessons can be learned from those.

Part 1: Internal vs. External success

Perceptions of risk depend heavily on company culture, with fear of failure (individually or corporately) being a major driving factor. Most companies agree that a reputation for secure, high-quality products is important to achieve market and business success. 

What differs is how success is measured: internally or externally. 

Companies with an internal focus generally use key performance indicators (KPIs) and other internally-defined metrics to provide incentives or disincentives to drive desired employee and departmental behaviors. These metrics are proxies for desired outcomes; for example, “minimize the number of security penetration vulnerabilities detected post-ship.” 

Causes and risks of an internal focus

Many large companies, in particular, see proxy metrics as the only way to effectively communicate goals throughout a complex organization, to achieve alignment, and to measure success business function-by-function or department-by-department.

Unfortunately, this focus on modeling desired outcomes using proxy metrics often results in the individuals within the company seeking to maximize (or minimize) the metrics that apply to them, rather than focusing on the overall success of a given product in the marketplace. 

No one in this system has the wrong intent. Who could argue that it’s good to ship with more security vulnerabilities? By seeking to eliminate security vulnerabilities, the security group (for example) is merely seeking to do its job. With an internal focus, they will see their job as ensuring compliance to the policies the team has established to achieve the goals set by upper management. 

However, a department-by-department optimization approach can lead to a bad outcome for the company. It is impossible — or at least economically infeasible in a finite time — to create a product of any kind, software or hardware, that is literally perfect in every dimension. And even in those areas where the product is “perfect,” time:

• exposes new patterns of usage
• introduces new vulnerabilities in underlying infrastructure or integrated systems
• results in changes in the operating environment
• spreads new usability paradigms
• and introduces other factors that will make even an initially ‘perfect’ product imperfect. 

Successful products are a balance of features, time to market, and non-functional requirements like usability, security, quality, and many others. Which aspect of a product should be emphasized at a given time to achieve a given business outcome requires balance across multiple factors that are sometimes — in fact, often — in conflict. 

In an internally-focused company, ship decisions are often made by executives based on the input from the various business functions and departments. Each of those stakeholders is trying to follow corporate direction by maximizing (or minimizing) the appropriate metrics. 

This can result in some groups having an incentive to avoid shipping a product at all, in order to minimize their department’s or their own exposure to a potentially negative impact on their group’s KPIs. And sometimes, not shipping is the right thing to do. 

On the other hand, in an internally-driven culture, a product that might otherwise be highly successful in the market might never be seen as “good enough” by all its internal stakeholders because shipping it poses a risk to a particular group’s KPIs or other departmental incentives. No one is doing anything ‘wrong’ or ‘dishonest’ in this scenario;in fact, the groups are responding to the direction they have been given by upper management in the form of these metrics. 

However, the net effect is that such companies tend to be highly conservative. They tend not to ship innovative products because they pose a risk of failure to meet internal metrics for one group or another.

How externally-focused organizations differ

Externally-focused companies, on the other hand, are market-focused and look for their signs of success or failure from their buyers, customers and end users. They, too, are sensitive to the risk of failure, but it’s commercial failure in the marketplace that keeps them up at night. 

No single group can afford to place its individual departmental success or mission above the goal of shipping a successful product. This is not out of altruism, or because the individuals involved have different skill sets, quality standards or personalities than their counterparts in an internally-focused company. Rather, it’s because of the way success is measured. 

Recommended reading: Culture at the Core: A Playbook for Digital Transformation

In an externally-focused company, the whole team — all departments — succeed or fail based solely on the success of the product in the market. No department is viewed as successful, even if it hits its individual metrics, if the company or the product fails. Market success is a shared goal, and all teams in an externally-focused company have no choice but to work together to achieve it.

That doesn’t mean that teams within a market-focused company always agree with each other. Far from it. 

Product Management and Engineering may disagree on dates and feature sets; the quality and security groups may raise red flags that are heeded or ignored; DevOps and Ops may fight with the business over tool choices and FinOps. What is different in an externally-focused company, though, is that all of these disagreements are in pursuit of a common goal rather than a departmental goal. 

That shared goal is to quickly ship a featureful-enough, secure-enough, sufficiently high-enough-quality product that succeeds in the marketplace. In this context, maximizing the metrics of an individual group is irrelevant if the product fails, because all lose. 

When all stakeholders are focused on a common goal, disagreements tend to be healthy and rapidly resolved because all are in pursuit of a common goal: product success.

Small companies, and startups in particular, tend to be externally, market-focused. The major incentive and KPI at a startup is stock options, and these only become valuable if the product and company succeeds in the marketplace.

Differences in focus between small and large companies 

Small companies tend to have only a small number of products — sometimes just one. If the product and company do NOT succeed in the marketplace, then a small company will likely close down, making the stock options worthless. Despite the market risk, these factors and others align to make startups externally focused (which is part of their attraction to many good engineers and investors).

Large companies may be or become externally focused, as well. Apple is a very good example. Before Steve Jobs rejoined as interim CEO in the late 1990s, Apple had become very much an internally-focused company. Jobs’ and Apple’s tremendous achievement was due in large part to Steve’s success at flipping Apple’s focus from internal to external. 

He did that by getting a critical mass of people focused on their product’s success with consumers, and in the marketplace. Jobs did this while maintaining what most of us would agree were high quality and security standards. “Externally focused” does not mean “sloppy.” 

Externally focused does mean “risk-based,” though. As a market-focused company, it does not make sense to put effort into areas that fail to generate business value. 

This means that instead of absolute, unalterable, KPI- or departmentally-driven standards for each individual aspect of a product, an externally-focused company instead looks at how best to maximize the business value of the entire product for each release. That is the essence of creating a product that is “good-enough,” “secure-enough,” “featureful-enough,” and so on. This does not mean ‘sloppy,’ but it does mean a ruthless focus on creating release-by-release business value.

Now, how do you quantify “good enough” when making risk-based ship decisions?

Part 2: Approaches to “Good Enough”

With this understanding of how market-focused companies make risk-based ship decisions based on maximizing the business value of each release, we can begin to explore different ways to quantify “how good is good enough.”

Achieving external market-based success often requires a balancing act. In general, you want to get a new product in front of real, live customers as soon as possible. This not only delivers value to your users, but also allows you to gather their feedback so you can improve the product (“pivot or persevere” in Lean/Agile terminology). Success here also lets you unlock potential revenue streams from investors, internally from your own company, and from customers themselves. 

On the other hand, if your product is not “good enough” when customers first see it, you may lose them irreparably and fail in a different way to unlock your revenue stream. In other words, there are risks in shipping, and also risks in not shipping. 

How do externally-focused companies make the tradeoff?

I started studying this issue back in the 1990s. At that time, I read a book that described Hewlett Packard’s quality criteria for at least some of their products at the time (this is pre-split, before HP became “HP” and “HPE”).  

The book said that HP’s quality criteria was met for a given product when it became more expensive to find the next critical bug internally through testing than the cost of letting the customers find that critical bug post-ship. In other words, they would ship when it was more costly to find the bug through testing than to let the customers find it, reputation and remediation costs included.

This was a very high bar in the ‘90s. In those pre-cloud days, systems were deployed on premises, on customer equipment. For at least some systems, replacing them in the field required flying technicians to multiple points of the globe to physically install from media, apply configuration options, transition data, and perform other operations. Replacing a critical system in the field was expensive, so this quality bar was very high. Today, of course, we can deploy in production automatically multiple times a day, so it might not seem like a big deal. But it was then.

Future events are always probabilistic; we don’t know for sure how much testing will be required to find the next critical bug, or how much customer use will uncover it. We can use tools like the defect discovery rate and projections based on past releases to put numbers on these events. But in the end, we are balancing one uncertainty or “risk” against another. This 1990s era HP criteria was therefore “risk based;” that is, they shipped when the probabilities dictated that it was likely that it had become more expensive to continue testing, than the likely cost of finding and remediating a serious quality issue found after shipping. 

I followed HP’s example with a number of products in that timeframe, and for many of them I was accused of being too conservative as the customer never did find that next critical bug. Still, we were playing the odds and very conscious of it.

HP’s approach appealed to me because it made sense, business-wise and quality-wise. You ship when it’s more costly not to ship. 

The potential cost of shipping could be defined so as to include all the factors that keep people up at night: security incidents, production-down incidents, reputational damage — even loss of life. These would be weighted by their probability of occurrence. The cost of not shipping, on the other hand, would include:

• loss of revenue, 
• loss of opportunity and competitive advantage, 
• the cost of continued development and testing, 
• development infrastructure costs, 
• and everything else that goes into keeping a product under on-going development without a supporting revenue stream.

Steve Jobs is quoted as saying to the original Apple MacIntosh team, “Real artists ship.” What he meant by this, I believe, is that software systems (or anything else) must be put in the hands of end users to be valuable to anyone. A system delivers no business value while it is under development. 

A system that never ships, because its developers or other stakeholders believe it’s imperfect, delivers no value at all. And, of course, no system is ever perfect. Even if it did start out being perfect, a real-life system is unlikely to remain perfect given emerging security threats, changes to the underlying software components a given system depends on, and many other factors outside the boundaries of the system itself.

Recommended reading: If You Build Products, You Should Be Using Digital Twins

Years ago, I was lucky enough to work with some PhD-level experts on economically-based business decision making. One then-widely accepted approach to such decision making was probabilistic and risk-based. I was surprised and a little shocked to learn that this often included putting a dollar value on lives lost, in order to calculate potential risk. 

For example, if an airline company wanted to determine whether a potentially life-saving improvement to one of their jets was worth the investment, they would go through a thought process something like this simplified illustrative example using fictitious numbers:

• Replacement cost of the plane is $100M.
• The plane carries 200 passengers.
• Our expected liability for each passenger death is $10M USD (this was in the early 2000s, so today this figure would be closer to $20M USD per passenger).
• The probability of a fatal crash over the 150,000 flight-hours expected service life of the airplane is 1.78%.

Using these made-up figures, we see that the “expected loss” (probability of loss times the $2.1B potential amount of loss) is $37M USD over the life of the aircraft. 

In other words, if you were going to set aside a reserve to cover future losses from the crash liability for “N” number of airplanes of the same type, you should set aside “N x $37M USD.” If the lifetime probability of loss could be reduced to zero through an investment of $37M per plane or less, then it would be economically justified to spend it on improving the safety of the aircraft. 

If it costs more to reduce the risk to zero, or in general if the amount at risk would be reduced by less than the amount invested, the additional investment to improve safety would not be justified in purely economic terms.

This is a very simplified analysis and perhaps morally repugnant because human life is involved. How can one put a financial value — even double the original value, or $20M in today’s dollars — on an individual human life? 

Yet the alternative is to never ship. If we had to reduce the risk of a fatal airplane crash to zero, no one would ever be able to fly. 

The cost of creating such a plane would so far exceed the potential losses from the current generation of planes that no company (or group of companies, or (probably) nations) would ever be able to make the investment. And even if it were possible to build a perfect plane, external factors like deliberate malice, meteor strikes or other non-engineering factors would mean that some people will still die. The cost of the added investment in safety would also have to be passed along to the consumer, raising the price of travel for the end user — perhaps raising it more than the decreased risk would be worth even to the traveler themselves. 

It’s heartless, but if one wants the benefit, one needs to take the risk. The art is making the risk of loss smaller than the expected reward, to the company and to the customers, who follow their own risk/reward calculus.

We each make risk-based decisions every day, yet we hardly think about them. We risk our lives, to a greater or lesser extent, every time we commute to work. For those who drive in the U.S., the cumulative lifetime risk of dying in a car accident is about 1 in 93. Yet we continue to drive and take other risks, because we believe the benefit outweighs the risk. We also put the lives of our loved ones at risk each time they accompany us in a car, ride a bike, or even go for a walk. We rarely worry about it or think about it, but at some level our brains do the math and decide to take a risk to get a reward. 

Even following a risk/reward model, we can still set the quality bar as high as we like by making the expected cost parameters as high as we choose. Instead of $10M liability per life, we can increase it to $100M or $1B per life, for example. We can continue to invest until an arbitrarily high degree of perfection is achieved — or until we run out of money. However, in the meantime, both the customer and the company are denied the benefits they would get through real-life use of the product. This is also a very real cost. 

We can safely assume you believe your product will deliver value to its users: If you didn’t think so, why would you build it in the first place? If that is your belief, then withholding your product from the market hurts its potential users by denying them the benefits that using your product would bring. While you obviously don’t want to ship them something bad or that doesn’t work, bringing something to market that delivers positive benefits to its users is valuable to them, and therefore worthwhile. 

Considering only the potential risks associated with shipping a product, without weighing the potential benefits internally and externally, is not a sound or a balanced ‘engineering’ approach to ship decisions. Weighing both costs and benefits when making a risk-based ship decision is not a compromise. It’s the essence of a value-based business and engineering approach.

Learn more:

• Software Engineering in the Age of Creative AIs
• AI’s Impact on Software Development: Where We Are & What Comes Next
• Software, the Last Handmade Thing

I was talking with my daughter, a creative and a commercial writer, about ChatGPT recently. At a client’s request, my daughter has begun using ChatGPT for some of her commercial work. She’s also started to experiment with AI-based image generation tools as a means for creating cover art and illustrations. She is often amused by what ChatGPT comes up with, but it’s clear to me that the potential is there for it to become a serious creative tool in diverse fields in the near future. This got me thinking… 

Could brilliant late 16th/early 17th century poet and playwright William Shakespeare have been replaced by an AI? 

Or might Shakespeare have used an AI, if he had one, as a creative tool? To experiment with this idea, I started with a sonnet (sonnet 18) actually written by Shakespeare himself somewhere in the late 1500s and first published in the year 1609:

Shall I compare thee to a summer’s day?
Thou art more lovely and more temperate.
Rough winds do shake the darling buds of May,
And summer’s lease hath all too short a date.
Sometime too hot the eye of heaven shines,
And often is his gold complexion dimmed;
And every fair from fair sometime declines,
By chance, or nature’s changing course, untrimmed;
But thy eternal summer shall not fade,
Nor lose possession of that fair thou ow’st,
Nor shall death brag thou wand'rest in his shade,
When in eternal lines to Time thou grow'st.
    So long as men can breathe, or eyes can see,
    So long lives this, and this gives life to thee.

On the assumption that Shakespeare was writing about a young person (which I think is a reasonable guess from the content of the poem and the fact that Shakespeare was 36 himself in the year 1600), and that he was writing about a woman (not at all certain, as some closely related sonnets were clearly written to a man — but I wanted to make my request to the AI as concrete as possible so I had to pick), I asked ChatGPT to “Write a Shakespearean sonnet comparing a young woman to a summer's day.”

Recommended reading: Tips for Staying Relevant in the Age of Creative AIs Like ChatGPT 

ChatGPT Writes a Shakespearean Sonnet

Here’s what I got back:

Oh fairest maiden, like a summer's day,
Thou art a sight that doth my heart beguile,
Thy gentle beauty, like the sun's warm ray,
Doth light my soul, and make my spirit smile.

Thou art more lovely than a summer's morn,
When golden light doth gild the earth below,
And larks do sing to greet the day reborn,
As thou dost grace the world with thy sweet glow.

The summer's day, it doth too quickly pass,
But thou, fair maiden, shall forever shine,
For in my heart, thy beauty shall amass,
And make my world a summer ever fine.

So let me bask in thy sweet radiance bright,
And be the moon that mirrors thy delight.

Taste in poetry differs greatly but from my personal perspective, the AI-generated sonnet is actually reasonably good. Keep in mind that my skill set is in science, math and engineering — not literature. 

The rhyme scheme (day/ray, beguile/smile, etc.) seems mostly unforced (though I’d question (pass/amass), and while somewhat disjointed, the poem flows reasonably well from idea to idea. The imagery seems appropriate, comparing the loved one’s beauty to sunshine, birdsong at sunrise, and to the warmth of the day. The poem even captures something of Shakespeare’s original nuance that summer is brief, and that beauty can be, too. 

But comparing it with Shakespeare’s original, it’s clear that even though Shakespeare claims in the first line of Sonnet 18 that he’s comparing his love to a summer’s day, in the subsequent lines he actually contrasts the two (“thou art more lovely and more temperate” [emphasis added]). 

Refining Our Ask of the Creative AI

So, let’s try again by asking ChatGPT to “write an original Shakespearean sonnet contrasting a young woman and a summer's day.” Here’s how the AI responded:

Thou art more fair than any summer's day,
With beauty that doth put the sun to shame,
For though the day may bring its warm display,
Thy loveliness outshines its brightest flame.

The sun may rise and set without a care,
And scorch the earth with its unyielding heat,
But thou art gentle, with a heart so rare,
And bring'st a calmness that is oh so sweet.

The summer's day may come and go with ease,
But thou art constant, with a heart so true,
Thy beauty brings a sense of perfect peace,
And fills my heart with love that's ever new.

So though the summer's day may fade away,
Thy beauty shall forevermore hold sway.

This, I think, is somewhat closer to Shakespeare’s original intent. In the closing stanza, the AI-generated poem touches on the idea that while summers are brief (especially in Shakespeare’s native England), his love will stay beautiful forever. However, it does not state the reason for this, as Shakespeare does in his original: namely that his love’s beauty is captured forever in the poem itself (“So long as men can breathe, or eyes can see,/So long lives this, and this gives life to thee.”) 

Okay, Perhaps Our Creative AI Needs More Guidance

Let’s try one final time by asking ChatGPT to “write an original Shakespearean sonnet contrasting a young woman and a summer's day, emphasizing that his love's beauty is captured forever in the poem itself.” Here’s what the AI came up with this time:

Fair maiden, thou art like a summer's day,
With beauty that doth strike the heart so true,
But though the day may soon slip away,
Thy loveliness shall live in this poem anew.

For as the sun may set and leave us cold,
And summer's warmth may soon be but a dream,
Thy beauty, in this verse, shall still unfold,
And like a flower, forever in bloom, gleam.

The summer's day may be but fleeting bliss,
A momentary warmth, soon to be gone,
But in this sonnet, thy beauty shall persist,
And shine eternal like the brightening dawn.

So though the summer's day may fade from view,
Thy beauty, in this poem, shall be ever true.

In my definitively-not-an-English-major mind, I think this AI-generated poem is fairly close in its intent to my admittedly distinctly amateur understanding of Shakespeare’s original. 

Technically, the poem is not as good as Shakespeare’s, in my opinion; for one thing, some of the language construction is a bit forced. For example, the word “gleam” chosen to rhyme with “dream” only makes sense if it refers to “Thy beauty” in the previous line, since flowers do not “gleam.” It’s not incorrect to construct a sentence this way, especially a stanza in poetry, but it’s awkward and somewhat confusing, again in my layman’s opinion. Also, some of the line endings seem chosen simply because they rhyme (true/anew, cold/unfold) and don’t add nuance to the poem itself. 

On the other hand, Shakespeare’s rhyming word choices sometimes reinforce and contrast with each other (shines/declines, fade/shade), adding a further dimension to the poem. Also, I find Shakepeare’s phrases (“rough winds,” “darling buds of May”) more vivid in the imagery they invoke, personally. But I think the major ideas that at least I myself take away from Shakespeare’s original are reflected in the AI-generated sonnet in some way, and the piece itself is basically serviceable as a poem.

Even if you agree with me that it does, basically, embody the same concepts as Shakespeare’s sonnet, would this AI-generated poem live on and inspire people for over 400 years, as Shakespeare’s has done? 

Part of the reason for the original Shakespeare sonnet’s fame are certainly cultural and historical. We were taught it in school and told it was good, so we believe it is good. We are also very familiar with the Shakespeare original, so it has the allure of an old friend, and resonance from the various times in our lives we encountered it before. It has the “first mover” advantage where you probably had already read it, in most cases, before you saw the AI-generated version. The works of Shakespeare also shaped the English language, to some degree, so our standard of measure is to some extent biased to make us appreciate the original.

Even doing my best to account for these factors, though, I have to say that the AI-generated poem does not compare favorably to Shakespeare’s original, and will not stand the test of time in the same way. 

If I had encountered the AI-generated version before I read the original Shakespearean Sonnet, I might have been excited by it. However, that excitement would not be because of the way it is expressed but rather because of the novelty (to me) of the thoughts behind the poem: namely, the idea to contrast “my love” and a “summer’s day,” and that unlike a summer’s day, that his loved one’s beauty will not fade because it is immortalized by the poem itself. 

However, we fed those thoughts to the AI in our original request; these ideas were not generated by the AI itself. What the AI did was to express those ideas, adding concrete illustrations, and phasing the result in a poetic form.

Original Thinkers Have Nothing to Fear of the Age of AI

This in no way lessens the achievement here by ChatGPT— what it can do is truly remarkable. That it can generate poems this good in seconds, on demand, and actually expressing the thoughts we gave it, is absolutely mind-boggling. Imagine giving the same instruction as an assignment to a human being. What would we get back, and when (if ever)? 

Of course, Shakespeare’s original sonnet (and all his works) were certainly ingested and processed by the AI, so it’s no coincidence that the original would inform the AI’s output in some way. We can’t hold that against the AI, however; nearly every human 21st century poet in the west — and many from other parts of the world — would have the same advantage of having read Shakespeare. Even given that, would most humans do as well?

Our conclusions from this exercise:

I don’t think Shakespeare or any original thinker has anything to fear from AIs. 

An AI is a potentially powerful tool that can help thinkers express original ideas in useful ways, and to help them generate new ideas as well.

A major driver behind many written, artistic and other works is not originality for its own sake, but rather to express what the author wants to say to a particular audience. 

Other people — or AIs — may give expression to those thoughts and feelings without taking away from the author’s intent. However, wherever they originated, when the written (or artistic expression) of ideas or feelings reflect one’s own, conveying that congruence can itself have value, depending on the audience. 

For example, providing an expression of my thoughts and my feelings to my loved ones can be significant to them even if I worked with an AI (or bought a greeting card, or bought a work of art or a gift) that expressed precisely what I wanted to say.

While ChatGPT and other AI systems in the pipeline are starting to usurp some of the functions that we thought of as human — creating art, poetry and computer programs, for example — the concept behind “delegating” creativity is nothing new. Executives often tell their assistants or their marketing departments the broad outlines of a communication that will later go out over their name. The actual content will then be written by an assistant, or by marketing, HR or another group. The fact that celebrities use ghost writers to produce works published under their own name is well known. 

In our everyday lives, we often search for a greeting card written and drawn by someone else that nonetheless conveys what we wish we could say (as poetically or beautifully) to a loved one. We buy art and other gifts created by other human beings that convey some feeling or idea we would like to express or remember.

Are truly creative people threatened by the fact that there are other creative people in the world? Maybe the insecure ones are, until they gain more confidence. 

But in general, I think creative people are more likely to be spurred on by the other creative people around them. Shakespeare, for example, is thought to have become a better writer because of his contemporary writers — and sometimes rivals and critiques — Ben Johnson and Christopher Marlowe. Having another creative person to bounce ideas off of, or to challenge your own ideas, is often the path to excellence. 

I think creative AIs like ChatGPT will assume that role: they will make us more creative, and better at what we do. I think Shakespeare might have used something like ChatGPT to help him get started generating new concepts, to do research (he wrote a lot of history), and to look for typos and grammar issues in his own work, for example.

And for those areas where creative AIs can do better work than a given individual can (I'm a terrible graphic artist, for example), they can help us to express our ideas and feelings in ways we can imagine, but otherwise could not bring to life unaided. 

Fearing a new source of creative expression is about as futile as living in fear that the person in the next desk is smarter than you, or a better coder, or a better writer, etc. Where that is true, your best strategy is to have the confidence to enlist their help — or to emulate their example and learn to do better. Where it is not true, your best strategy is to find what you do uniquely well, and partner with your office mate to produce something really great. 

AI or person, learning from it and partnering with it is what I think Shakespeare would have ended up doing with a technology like ChatGPT. Or, as Shakespeare himself might say: "To use AI, or not to use AI: That is the question."

More helpful resources:

• AI's Impact on Software Development: Where We Are & What Comes Next
• If You Build Products, You Should Be Using Digital Twins
• Complex Digital Transformation Programs [Ebook]