When KPIs are Blinders: The Dangers of Local Optima

/in , , /by

Recently we encountered a strange situation.  One of our clients had a top of the funnel KPI that looked like it was going sideways with regards to efficiency, but their revenues were going gangbusters. We use this metric for optimization and it is the core number by which everyone from their Board of Directors on down judge the efficiency and success of the digital campaigns.

Suffice it to say, this was not good.

Interestingly enough, the revenue numbers were setting new records, which made us question what was happening with our long standing and go-to KPI. So we started checking off the boxes. Was the revenue coming from the current marketing investment? Check. Were we paying what our financial models said was appropriate to the audiences? Again check. Was the top of the funnel KPI consistent over time and audiences?

Hmm. No, no it wasn’t. Aha! Changes to user engagement paths had affected the top of the funnel over time, and the average KPI from the past was no longer aligned with revenue and profit.

Our go-to KPI had become a Local Optimum, and we needed to address it quickly.

I first came across the term Local Optima in Eliyahu Goldratt’s classic novel on business optimization The Goal.  Essentially, a local optimum is a metric that may apply to a “neighborhood” of criteria, but is uncorrelated with the entire system. Goldratt used the uptime of a single machine in a factory as his example, where running any machine at 100% capacity other than the one machine that throttles throughput for the entire factory will cause inefficiencies and losses for the entire system.

Goldratt and his Theory of Constraints thinking dictate that “Measurements of local optimum behavior should be abolished and replaced with holistic measurements.”  If we had a Local Optimum on our hands, attempts to “correct” for this KPI would not be mildly bad, it would almost assuredly hurt the profitability of the entire company.

So we threw out the KPI and went back to first principles: “Marketing is a financial investment in a financial outcome”. What could we look at that would be correlated to revenue production over time?

It turned out that slightly down-funnel was another metric that turned out to be 100% correlated to revenue production and, in fact, was the one to which we calibrated our bid models.  We were able to show that the sideways direction of the top-of-funnel KPI wasn’t a problem, but was actually a natural side-effect of using the “true” KPI to back-calculate to the top of the funnel. By shifting the conversation from our most visible KPI, the conversation then quickly shifted to how to take advantage of this knowledge. How could we change our reporting, conversation, and client-side metrics to align? We lucked out. Our client is extremely savvy in understanding their own cohort data and financial targets and rapidly escalated the conversation to the C-suite and their Board.

We talk consistently about using Profit as a KPI. In reality, as digital marketers we often have to use early engagement numbers as proxies for true Profit. It is always good to step back and assess the relationship to Profit in case a Local Optimum has snuck into the room as it did in this situation.

What Magritte Has to Tell Us About Marketing Data

/in , , , /by

In the winter of 1928-29 a Belgian painter living in Paris painted what looked like an advertisement for pipe tobacco. The painter, René Magritte, painted a caption below the large pipe stating “This is not a pipe” creating confusion for viewers and a bit of a stir in the Paris art scene at  the time.

The title of the painting is “The Treachery of Images” and Magritte intended it as a reminder that representational art is a reflection of the artist’s view of the object,  not the object itself. His interpretation was that  the painting of the pipe is not actually a pipe. You can’t smoke it, fill it with tobacco, or put it in your pocket; it is a representation of reality.

Magritte’s message is particularly apt in our view of marketing data. As marketers, we use our marketing data all the time in measuring and assessing user behavior related to digital advertising and engagement. But it is extremely important to remember that marketing data is an imperfect representation of user behavior and not a perfect simulation.

There are three main reasons that marketing data is imperfect. The first is in the nature of tracking. Tracking is technically limited in its scope and reach.  At best, tracking can measure engagement from the same device over time, or the same multi-device account over time. This only works if the tracking is implemented correctly in the first place and is not disabled by the end user. Because of the technical limitations of tracking there will always be engagement that is not tracked because of multi-device use, the amount of time for which the tracking is active (cookie window), or because of personal opt-out at the user level either by choice or by browser pre-set settings.

The second main source of imperfection in marketing data is in user behavior.  As humans we have a wide variety of choices and methods of interacting with advertising. Some of us choose to avoid interacting with advertising as much as possible while some of us behave in the opposite fashion. Some of us will choose not to click on ads at all, some of us don’t hesitate. Some of us need to heavily research purchases, some don’t. There is not one engagement path that is adhered to by all users yet often our data is interpreted through a “single funnel” lens that introduces inaccuracies in interpreting data.

The last main source of data imperfection is in data integrity. The cleaning and analyzing of data must align with the knowledge being sought. De-duping rules will differ by business model. Attribution will vary based on external factors such as affiliate payment rules (and will still never tell a 100% accurate story). Data dropout can cause interpretation issues.

So what is a marketer to do?

The best thing you can do is to fully utilize data to test your own model of reality.  The best data firms don’t blindly use data but instead use it to inform their perception of reality.  For example, if you know that a percentage of social users engage with your brand without clicking on ads and instead show up as brand search or No Referrer traffic, you should try to assess: how much, how can that magnitude be assessed and what impact does it have on my decisions as a marketer if I can hypothesize about non-measured (but real!) activity.

Magritte’s point  is well-made. Our data is not reality, but an imperfect reflection of reality biased by user behavior, technical limitations, and process. Our best response is to acknowledge and accept this and use it to our advantage.

Why ROAS Sucks As a KPI

/in , , /by

ROAS (Return on Ad Spend) is the industry standard for assessing and reporting the performance of paid digital campaigns, but it really sucks as a KPI.

To be totally fair, ROAS has a few strengths. It is easy to calculate (even easier than ROI): Revenue / Ad Spend. It is a relatively clear measure of efficiency. It is a metric that can be used across channels and media. But that’s about it, and when you start digging in, ROAS has a lot of limitations, including one huge killer that makes it, well, dangerous:

You can go out of business optimizing to ROAS.

And that is where ROAS really comes into its own as a sucky KPI. Frankly, I want KPIs that align with the financial health of my business and I certainly want that alignment in KPIs used for assessing campaigns we run for clients. ROAS doesn’t do that. ROAS is an efficiency metric. It has very little to do with how much money you make. This is because (like ROI) it is only a good metric when comparing exactly the same media spend. The problem is, nobody does that.

When you look at marketing reports across the industry, you will almost always see ROAS as a standard KPI, even when marketing spend changes.  And this is where the danger part comes in. ROAS works as a comparison of efficiency, and not magnitude.  An ROAS of 6 is more efficient than an ROAS of 5, but that doesn’t mean you made more money (or any money, but I will get to that). An ROAS of 6 on $10 did not make you more money than an ROAS of 5 on $10,000,000. That is a fairly obvious example, but marketers often treat ROAS as a KPI in isolation, which means you can easily reduce the magnitude of returns in the search for higher efficiency. Google even allows campaigns to use ROAS as a target for bid optimization, regardless of the effects on magnitude. Ponder that for a moment. And while it seems crazy, we have seen marketers make the decision to optimize for ROAS, opting for efficient returns even when the magnitude wasn’t enough to make payroll and keep the lights on.

Which brings us to the second danger with ROAS which is that it is based on top-line revenue, and not profit. This is why you can have positive (>1) ROAS and still lose money. This can happen when the efficiency of the return isn’t enough to cover costs. Unfortunately, many marketers aren’t given deep enough financial information to know whether campaigns are profitable or not. They just have this one metric which will only ever tell them if campaigns are so grossly inefficient that they aren’t even covering revenue. Again, I want metrics that alert us when we are even slightly less profitable, and not just when the bus is so far off the road that it is lying at the bottom of the canyon.

So what’s a good metric? Total post-marketing profit (aka Contribution Margin), which I will cover in an upcoming post.

5 Marketing Data Mistakes Most Companies Make

/in , , , , /by

Even the most data-savvy of marketing teams can make mistakes in thinking about the use of data in optimizing campaigns for financial success. Let’s face it, marketing data is rife with issues and is never perfect, and it is easy to put on blinders based on the data you have available, the systems you use, the marketing channels you work with, or even the directives of senior executives. Despite that, everyone wants to be able to connect the dots from ad impression to profit. Here are a few common data traps we see even smart companies easily fall into. How do you rank on this list?

1. Using Average Value CPA Targets

Cost-Per-Acqusition targets are fertile ground for data issues. For example, are your targets even based on customer value, or are they merely a “seems reasonable” guess? Smart data-centric companies will base target CPAs on actual customer value to ensure that their marketing programs don’t risk over-paying for customer acquisition. However, even smart companies can fall into using a single average CPA target for all their customers. In doing so, they underpay for audiences that provide higher-than-average value customers and overpay for low-value customers. Companies can avoid this by using targets tied to segmented customer values, not averages.

2. Using Last-Click Attribution

The attribution question has long been mired in a false discussion of “who gets credit?” when there is more than one user touch leading to a sale or lead. Some companies still use last-click attribution, often in a mistaken belief that this is somehow a “truer” view of acquisition, or that it just allows them to avoid thinking about attribution at all. Google hasn’t made things easier by offering multiple views of attribution with little guidance on when and where the different options should be used. Here is our take: Avoid last-click attribution at all costs unless you are evaluating retargeting assists. Last-click attribution will severely over-inflate your brand and direct numbers and cloud your ability to see high-value first-mover channels.

3. Not Considering Out-Of-Channel Effects

Everyone looks at their channel-specific numbers, but an astonishingly few companies continually examine their direct and brand channels, looking for influence from other areas. While everyone logically understands that users did not wake up with magical knowledge of a company’s brand, it often feels like that’s the assumption of marketing teams who put on “channel blinders” when evaluating their programs. Smart companies view their data holistically, looking for out-of-channel trends that increase or decrease direct and brand engagement. While only 5% of users in a typical search campaign are likely to bounce over to direct or brand, it is not unusual for a whopping 50%-90% of sales from social media or display campaigns to come through direct or brand traffic.

4. Over-Valuing Metrics Not Tied to Revenue

What is the value of a Like? Most data-driven marketers have moved on from directly equating social media engagement as revenue-related events, but many metrics that don’t correlate well to revenue are still held as sacred cows. Any metric used for campaign optimization should be well understood in how it relates to revenue before it becomes a key KPI. Data-driven marketers with their eyes on the profit prize quickly realize that Time-On-Site, Impression Share, Cost-Per-Click, or other common metrics are not as tightly aligned with profit as they might think when other factors such as volume, customer value, out-of-channel influence, or profit margin are taken into account. Easy rule of thumb: Use post-marketing profit as your marketing KPI.

5. Assuming Traffic Equals Sales

We’re two decades into the digital revolution, and it’s still incredibly common for people to assume that eyeballs equal profit. Back in the days of traditional media the best shot you could make in media buying was the most eyeballs for the lowest cost. That approach doesn’t work in digital because of the competitive auctions, and yet “Let’s get more traffic to this page/product/site” is not at all an uncommon marching order, particularly from executives who don’t understand the auction effects in digital media buying. Smart data-driven marketers know that their job is as much about when NOT to buy traffic as it is in finding the areas of success, and continually evaluating how to increase the quality of an audience by peeling away the “eyeballs” that are not their target audience. This allows them to compete more aggressively in the auctions while protecting the bottom line. Sometimes less really is more.

These are samples of marketing data traps that are very easy to find in almost any campaign. Most of these issues can be avoided through three core practices: 1) By adhering to a holistic financial lens in optimizing the entire marketing program against financial targets; 2) By working backwards through the path that led from advertising to revenue and; 3) By not ignoring revenue that falls outside of the “channel buckets”.

Attribution Lesson: Why Your Data Are Lying to You

/in , /by

Your data is not telling the truth, the whole truth and nothing but the truth. Well, the tools you’re using to manage your data are likely not up to the task and as a result of that attribution issue, you’re not getting to the truth.

In his first Search Engine Land article as a monthly contributor, Soren Ryherd encourages marketers to plug value leaks, measure campaign effectiveness with a holistic view, and to embrace the Out-of-Channel engagement path to help ensure that they are correctly managing their online marketing efforts.

Read this Search Engine Land article here!

What readers had to say…

“One of the best attribution articles I’ve read in a long time. Thanks for stepping up the conversation”.

“Very interesting thought on attribution ! You’ve inspired me to start thinking in a new direction”

What “The Next Level” Looks Like for Digital Marketing

/in , , , /by

“We want to take our marketing to the next level.”

We hear that often. But with digital marketing becoming more complex by the minute, what does the “next level” even look like?

1. It’s Holistic

With cross-device and cross-channel behavior becoming more the norm than the exception, marketing programs that are siloed by channel are going to be increasingly inefficient. For example, Mobile ad exposure can drive desktop engagement. Video ads may well drive brand searches.  But when value created in one channel is realized in another, it creates disconnects for budgeting and performance measurement. Striving for neat and tidy single-channel performance numbers is likely detrimental for your business. Marketers who break through to the next level are looking holistically at engagement across channels, across engagement points, and over time.

2. It’s Even More Data-Driven

Data-driven reporting, or simply looking at Google Analytics reports won’t cut it any more. Data used just for creating reports is data that is being wasted. True data-driven marketing feeds core metrics from advertising, engagement tracking, and customer databases back into ongoing optimization on a daily basis. “Next-level” data will focus on supporting complex modeling with hard data from multiple sources. The days of “do, measure, done” are gone.

3. It’s Financially Rooted

If you’ve been optimizing for marketing metrics without an underlying financial model, it’s time to step it up (take notice “Time-On-Site”). Marketing at its core is a financial  investment in a financial outcome. “Next level” marketing is explicit about the financial model and how engagement and customer value support revenues and profits. To do this, Key Performance Indicators must be tightly aligned with profits.

4. It’s Predictive

The ad dollars you spent today may have no relationship to the sales you made today unless you have the world’s shortest sales cycle. Today’s sales were likely to be largely, or maybe even entirely, from past advertising.  Tomorrow’s marketing relies on predictive models that are savvy about time. An understanding of sales cycle, latency and engagement process is critical to the financial assessment and efficient optimization of your program.

5. It’s Complex

Yes, life would be easy if marketing were simple. But today’s big opportunities in Digital lie in mastering complexity. The better you are at understanding complex user behaviors and tightly optimizing those behaviors for a clearly-stated financial result, the better chance you have to beat your competition. This means understanding the limitations of your ad management tools, reporting, tracking systems, and customer data and moving beyond those limitations.

The “next level” is an exciting place. Welcome to it.

To “Portfolio” or “Not Portfolio”?

/in , /by

Over the last few years there has been a lot of attention in the paid digital world over the “Portfolio” approach to paid search campaign management.  Some view this as way to maximize opportunity, while others try to draw analogies with stock portfolios as a way to manage risk.  Unfortunately, there is still a lot of confusion over what a portfolio approach is and whether it is actually good or bad.  And that is not surprising, as the definition of a portfolio approach changes depending on who you talk to.

Most portfolio approaches look to move above the keyword level of granularity in order to broaden audience exposure and to “pull out of the weeds” of keyword-level detail.  On the face of it, there is real value in doing this. It is common to have many keywords with very little engagement data. Assessing small data sets is tricky, so this broader view may make it easier to identify trends or performance numbers.  A broader view can also help in identifying performance issues in relatively short time periods, where, again, there may be little data at the keyword level.

But here is where it breaks down.

Some companies use Portfolio approach in a way where an average performance number of an aggregated number of audience segments is used instead of more granular data, even when the more granular data is actionable.  For us, the word “average performance number” is a huge red flag.  Audience segmentation is at the heart of gaining efficiency in digital campaign optimization, but this type of Portfolio approach goes in the opposite direction. This can lead to a very dangerous effect: subsidized performance.  Subsidized performance is when a poorly performing audience is masked by a strongly performing audience through the use of an average performance number.  Want to make your non-branded campaigns look good?  Fold their performance in with your branded campaigns.  Want to make PPC look good?  Fold in the organic performance (and sales) numbers.  Subsidized performance lurks everywhere, and it is important to be able to pick apart the average value to improve performance.

So, is Portfolio a bad approach?  Our take is that one should proceed with caution.  We work at the level of actionable data, which means that what we might call a Portfolio will change depending on how we are using the data, so we don’t tend to describe our approach as Portfolio. Since many Portfolio campaign structures (in bid management tools, for example) tend not to be picked apart for optimization purposes, and what constitutes a individual portfolio cam be very subjective, we’re wary of canned approaches described as Portfolio.  A more fluid approach that works at the level of actionable data will be far more useful in avoiding subsidized performance and finding true gains in profitability.

Bidding on Your Brand

/in , , , , /by

Bidding on brand terms in search campaigns is hotly debated.  In one camp are those who argue that these are cheap conversions, so of course you should bid on them.  In the other corner are those who say you will get those conversions anyway through organic, so it is simply a waste of money.

They are both wrong.

You should bid on your own brand terms in order to raise conversions.

Brand terms are the end of a conversation that began somewhere else.  No customer wakes up suddenly aware of your brand, ready to search for you. Some education has happened, somewhere, at some point in the past.  If you are using multi-touchpoint tracking, you might have some insight into the attribution chain leading to that brand search, but often that will fail to provide any insight (if, for example, someone did research at home and searched at work).

At this point, you may not know anything about this prospect other than that they have heard of you and have some interest in your company and products.

What are you going to say to them?

Most companies have multiple messages that speak to their value proposition.  Some will be feature-based, most will speak to benefits.  How sure are you that your organic listing includes ALL the important messages for every audience that might lock in the sale?

Bidding on your paid brand terms gives you the chance to say something different.  You can alternate your message, say something new, provide an offer, or reinforce an ad campaign that might be driving brand activity, such as display, video, or traditional media.

But the important thing is to look at conversions.  If you are investing dollars, you need to get additional value out of it to justify the cost. For brand advertising, this should be measured in increased conversion rates, viewed holistically, with an eye to what is driving the brand conversation in the first place.

As always, it is not about clicks, CPCs, or competition.  It is about profit.

ROI and other Dangerous Metrics

/in , , /by

Recently, I answered this question on Quora: “If the cost of PPC is £10, what is the ROI?”

A simple question, right?  But assessing ROI for PPC campaigns has hidden dangers, so I used the opportunity to not just answer the ROI definition question, but to point out some common misconceptions in optimizing to a pure ROI metric.  Here’s how that went:

ROI (Return on Investment) has a specific definition, but is often used fairly loosely to mean the profitability of a company or program.

Technically,

ROI = (Revenues – Costs) / Costs

So, If you made £20 from your £10 investment in PPC, your ROI would be:

(20 – 10) / 10 = 1, or 100% ROI

This differs a bit from Return On Ad Spend (ROAS) which is a commonly used metric:

ROAS = Revenues / Ad Spend

For the same example: ROAS = 20/10 = 2

ROI is usually expressed as a percentage, and ROAS as a number. Note that an ROAS under 1 is unprofitable, while any positive ROI number indicates profitability.

Now, since you asked about PPC specifically, let me point out the DANGER in both of these calculations. So here is a question:

“Which is better, 100% ROI or 200% ROI? What about an ROAS of 2 or an ROAS of 4?”

Like most people, you are probably going to say “Well, Soren, that’s a dumb question, of course the higher number is better in both cases”.

And you would be right, as long as the cost number is equal in both cases.

The big danger with ROAS and ROI is that they are often used to compare situations that are not apples-to-apples with regard to cost. Volume is completely missing in these equations.

In PPC, optimizing to ROI as a percentage can put you out of business. Don’t believe me?

Here’s another question. “Do you want profit of $1,000,000.00 at 100% ROI or profit of $100.00 at 400% ROI?” No brainer. I want the $1M profit regardless of ROI.

The real goal is the total amount of profit, not ROI.

Unfortunately, this important point is lost on many CEOs, CMOs, and, surprisingly, CFOs when speaking to their marketing teams or partners. ROI can be a good measure of efficiency when used appropriately, but it is not the goal.

Case Study: Effects of Seasonality and Brand Traffic on Campaign Performance

/in , , /by

Client: Non-profit organization relying on monetary donations to fund core programs

Overview: Our client is a non-profit organization that relies on monetary donations to fund their mission. Their success is built on cost-effectively acquiring donations through in-kind, dollar, and airline mile donations programs via online and offline channels.  In analyzing their historical data and in advising on better tracking of donor activity, we found that there were substantial areas for increased efficiency in new donor acquisition through their paid online marketing programs.  In particular, a better understanding of non-brand keyword behavior and how to better understand the value of their strong and established brand as it relates to new donor acquisition led to dramatic breakthroughs in marketing efficiency.

Results:  The understanding of seasonality and differing donor behavior and donor value by segment has allowed us to dramatically increase the efficiency in acquiring new donors, and to best budget and execute to take advantage of highly seasonal trends.  These insights combined with efficient execution of the digital campaigns will yield significantly stronger results both during and outside their seasonally strong peak season and will allow them to more comfortably extend their digital programs into new areas.