This article is about the statistical and economic theory that underlies Recoset’s real-time bidding strategies, as a follow-up to my previous article on how we apply control theory to pace our spending, which I’m grandfathering in as “Part 0” of this short series.

At Recoset, we develop real-time bidding algorithms. In order to accomplish advertiser goals, our algorithms automatically take advantage of other bidders’ sub-optimal behaviour, as well as navigate around publisher price floors. These are bold claims, and we want our partners to understand how our technology works and be comfortable with it.  No “trust the Black Box” value proposition for us.

In Part 1 of this series I’ll explain the basics of our algorithm, first how we determine how much to bid, and then how we determine when to bid. In Part 2 I’ll show how this approach responds in some real-world situations.

First, Tell the Truth

Let’s say, for sake of illustration, that you are Recoset’s bidder.  You’re configured for a 3-month direct-response campaign with a total budget of $100,000 and a target cost-per-click (CPC) of $1.00.  Like every other participant in real-time auctions, you subscribe to a stream of several thousand bid requests (also known as ad calls) per second, each representing an opportunity to immediately show an ad to a particular user on a particular site. You must respond to each bid-request which matches your targeting criteria with a bid, within a few dozen milliseconds. How do you decide what amount to bid?

Most real-time auctions are what are known as Vickrey, or second-price, auctions.  The winner is the one who places the highest bid, but pays the amount of the second-highest bid. This type of auction was likely chosen by the designers of real-time exchange because the best strategy for bidders in such a situation is to “bid truthfully” (one of the well-known results of auction theory). This means that the bid you place should be equal to the amount that the good is worth to you.  In this case the “good” is the right to show an ad.  How much is that “worth”?

The target CPC is $1.00, so it’s not a bad assumption that a click is worth at least that much to the advertiser.  Let’s say that the average click-through rate (CTR) is 0.05% for this campaign: one out of every two thousand impressions gets clicked on.  So if you win this auction, you have a 0.05% chance of getting something that is worth a dollar, and multiplying those together we can say that the expected value of winning this auction is 0.05 cents.  If you’re bidding truthfully (which auction theory says you should), that’s how much you should bid.  One equation you could use to come up with a bid is therefore:

$$ bid = value = CPC_{target} * CTR $$

That’s a bit too simplistic, though, and doesn’t take advantage of the real-time nature of the exchange.  We’re a predictive analytics company, so as our bidder you have access to some fancy models.  These models predict, in real-time and on a bid-request by bid-request basis, the probability that this specific user will click on this specific ad in this specific context at this specific time. This means that you don’t have to use the campaign-wide average CTR when computing your bid, you can use the probability that our predictive model has generated for this request:

$$ bid = value = CPC_{target} * P(click) $$

So far so good, but not particularly groundbreaking: where is the secret sauce, other than in the probability-of-click prediction model which is beyond the scope of this article? That predictive model is certainly part of it, but there’s actually an additional wrinkle to the above equation: it only tells you what the bid should be if you actually decide to bid. And when deciding whether or not to bid, you shouldn’t just look at the value of winning the auction, but the cost as well.

To Bid or Not To Bid, or: Cost is not Value

As I talked about in Part 0, if you just bid on every bid-request that comes your way, you’ll spend the budgeted $100,000 way ahead of the 3-month schedule. In that article I also detailed a system for spreading your spend out over time by using a closed-loop control system to vary the probability of a bid being submitted.  Let’s try to take that idea further: what if instead of randomly selecting which requests to bid on, you could find a way to bid on only the best requests? I just showed a way to model the expected value of winning an auction but in this context “best” doesn’t just mean “highest value”; that’s only half the picture.

As an example of this, let’s look at an importer purchasing one of two types of goods for resale. Good A has a 10% lower resale value than Good B, but it costs half as much:

Two goods with similar values but different surpluses.

In this case the less valuable Good A is “better” because because what matters isn’t just the value or the cost but the difference between the two, or the surplus.  In business this is called profit and in game theory this is usually called the payoff.

$$ surplus = profit = payoff = value – cost  $$

The equation above holds for the RTB context if you win the auction, but what if you don’t? In that case, you get no chance at a click, so the value is zero, and you also don’t pay anything, so the cost is zero (modulo fixed costs, which are out of the scope of this article).  So for our purposes, the expected surplus is the same as above, multiplied by the probability of you actually winning the auction:

$$ surplus = (value – cost) * P(win) $$

Now assuming that alongside Recoset’s click-probability predictor, you also have access to a clearing-price-predictor (again beyond the scope of the current article) that tells you for each bid-request how likely you are to win the auction at each price, you can now compute the expected surplus.  (As it happens, one can also maximize this equation as a function of the bid to prove the well-known “bid truthfully” result mentioned above: the bid which gives you the highest surplus, no matter the price distribution, is equal to the value).

A surplus surface as a function of bid and value. For any given value, the maximum surplus occurs where bid=value.

So now when our closed-loop pacing control system says that in order to meet the budget requirements, you should only bid on a quarter of the request stream, you can choose to bid only on the bid-requests which have expected surpluses in the top quartile. This means that the control value is no longer the probability of bidding, but some measure of selectivity in the auctions you are willing to participate in: the lower the control value, the more selective you should be. You’re still placing the optimal bid every time, but now you’re specifically targeting the auctions where you think you will make the most profit given the target CPC, thereby increasing your chances of actually achieving a much lower CPC.

Algorithm Meets World

This post explains the two major components of our system: bidding truthfully and pacing economically. In Part 2 of this series, I’ll show how this approach to RTB handles some real-world situations advantageously.

In Part 1 of this series, I claimed that Recoset’s RTB algorithm is able to take advantage of other bidders’ sub-optimal behaviour and navigate around publisher price floor in order to achieve advertiser goals. I then described the algorithm, which applies what can be called a “bid truthfully, pace economically” approach. In this second part, I show how this algorithm can in fact live up to these claims.

When you bid truthfully and pace economically, you are always trying to allocate your budget to the auctions which look like the best deals, whether that means that the user is very likely to click, or that the price is low because fewer bidders are in the running or there is no publisher price floor.

How to Take Advantage of Other Bidders’ Behaviour

In Part 0 of this series, I showed that the average win-price for RTB auctions jumps up at midnight Eastern Time and described a pacing system which, if used by everyone, would prevent this jump, by spending at a constant rate. This system works by frequently updating a control value which is interpetted as the probability of bidding on any given auction. There is an implicit assumption underlying this system though: that any two randomly-selected bids are equivalent, regardless of, for example, the time of day. So while I introduced this system by talking about variability in the clearing price over the hours of the day, the simple pacer actually ignores this fact. This did not escape all readers of my post, some of whom commented in private that if we know the price is much higher at certain times, then we just shouldn’t bid at those times!

Now that I’ve described the way we compute expected surplus on a bid by bid basis, I can explain a principled way for a pacer to simultaneously handle variability in clearing prices and impression quality that doesn’t involve hardcoding hours when not to bid. All that is needed is for the pacing system to update the control value much less frequently, say once a day, and to treat the control value as a “minimum acceptable expected surplus” threshold. Consider the following graph comparing the evolution of average clearing-price and average expected surplus over a few days, noting how at midnight the price jumps and the surplus drops:

The evolution of clearing price and surplus over time.

A well-tuned, infrequently-updating pacing system will essentially draw a horizontal line across this graph, and our bidder will bid whenever the surplus is higher than this threshold. The key insight here is that the curves here represent the means of some fairly wide and skewed distributions. Even if the mean price goes up and therefore the expected surplus drops, that doesn’t mean there aren’t any “good deals” to be had: it just means there are fewer of them. By setting a hard, slowly-changing threshold, our bidder can continue bidding throughout the day, but it will probably spend less in the hour right after midnight than in the hour right before midnight.

We like to say that this is taking advantage of other bidders’ sub-optimal behaviour because some bidders are irrationally piling into the market right after midnight and driving the price up. Our algorithm continues to “skim the cream”, bidding in the best-looking auctions, but does back off a bit, holding its budget in reserve for such times as other bidders have run out of money for the day. As that happens, the drop in demand causes a drop in clearing price and a concurrent rise in expected surplus, causing our bidder to spend more.

Navigating Around Publisher Price Floors

Having covered how our algorithm responds to the actions of other bidders, how about publishers? How does this system respond to publisher price floors? A price floor, or reserve price, is basically a statement by a publisher that they will not sell below a certain price. It’s almost as if a publisher is bidding on its own inventory. If there are no higher bids than the price floor, there is no transaction. If there is more than one bid above the price floor then the price floor does nothing and the winner pays the second price. If there is only one higher bid, however, that bidder wins and pays the reserve price. In this third case, the publisher gets some of the surplus from the auction that the winner would have gotten had there been no floor.

This all makes perfect sense at the micro level, as by definition if we win the auction, we were willing to pay more than we did: our bid was how much it was worth to us, and we paid the second price so the difference between the two is profit for us. Any publisher would rather that we had just paid them the entire bid amount so they could get that profit, and therefore they have an incentive to raise the price floor all the way to our bid amount to “capture back” as much of that surplus as they can.

Our algorithm responds to a publisher price floor the same way it responds to the actions of other bidders causing the price to rise. At any given time, the pacer’s control value tells our bidder the minimum expected surplus required for it to bid. If a publisher raises their price floor (and if our price model captures this), the expected surplus of auctions for their impressions will drop. If it drops below the control value, we will simply not bid on those auctions at all, rather than accepting the higher price. This is what we call navigating around publisher price floors.

The control value is set so that the budget will be met, and is based on the availability of surplus on the whole exchange, not just on any one publisher’s sites. In effect, this puts an upper bound on the price floor any publisher can set before losing our business, so long as we can get a higher expected surplus elsewhere. This upper bound will be lower than the value to us of winning the auction, but could still be higher than the next-highest bid, so we do give up some of the surplus to the publisher. This is pretty much the way the exchange should work: it’s a market mechanism which uses competition to divide up the surplus pie.

The control value places an upper bound on the price floor that is lower than the value.

It’s worth mentioning that at the macro level, publishers don’t necessarily sell all their inventory on a spot basis on the RTB exchange: they try to sell inventory at a much higher price on a forward contract basis ahead of time. In industry jargon, inventory sold on a forward contract basis is called “premium” and everything else is called “remnant”. Without getting too deeply into this, publishers have an incentive to put up very high floor prices so that advertisers don’t stop buying expensive premium inventory because they think they can buy the same impressions more cheaply on the spot market (i.e. as remnant inventory). The industry lingo for this effect is “cross-channel conflict”, because the remnant “channel” is perceived to be hurting the premium “channel”. If a publisher is using this type of rationale to set a price floor, they will likely set it higher than the amount of our bid, rather than trying to squeeze it in between the first and second highest bids. This would translate into an expected surplus of zero, as there would be no chance of us winning the auction. Again, our algorithm just doesn’t bid on this type of auction; it certainly doesn’t raise the bid value to win an unprofitable impression. Mike on Ads has a good post about why it’s a bad idea for publishers to do this, based on some hypothetical behaviour of bidders. Our algorithm can automatically implement the types of behaviour he describes.

I’ve written some follow-up posts to this one, as a series called Peeking into the Black Box, so I’m grandfathering this post in as “Part 0″ of the series.

I read an interesting post on the AppNexus tech blog about their campaign monitoring tools and the screenshots there almost exclusively contained various pacing measurements. Some of the graphs there looked a lot like the ones I had sketched up while trying to solve the pacing problem for our real-time bidding (RTB) client.

Here are the basics of the problem: if someone gives you a fixed amount of money to run a display advertising campaign over a specific time period, it’s generally advisable to spend exactly that amount of money, spread out reasonably evenly over that time-period. Over-spending could mean you’re on the hook for the difference, and under-spending doesn’t look great if you want repeat business. And if you don’t spend it evenly, you’ll get some pissed-off customers, like this guy who had his $50 budget blown in minutes. Sounds obvious, right? Apparently it’s harder than it looks!

Doin’ it wrong

When Recoset first dipped its toe into RTB (real-time bidding, characterized in this sister post), the main tool at our disposal to control budget/pacing was a daily spending cap. It took about 48 hours for us to figure out that that wasn’t going to work for us: depending on the types of filters we used to decide when to bid at all, we’d either not manage spend the daily budget (more on that at the bottom) or blow the budget very quickly, usually the latter. In practice this meant that we would start buying impressions at midnight (all times in this post are in Montreal local time: EST or EDT) and generally hit our cap mid-morning.

No bidding occurs after daily cap is reached

On a daily spend graph over a 30-60 day period resulting from such a policy, this would probably look just fine: you’re spending the budget in the time given, and spending exactly the right amount per day. But having this sort of ‘dead time’ where we don’t bid at all bothered us, so as soon as we spun up our own bidder we added options like “bid on X% of the request stream” and when we looked at the resulting data we noticed something really interesting: the impressions we were buying before were actually among the most expensive of the day!

The average win-price graph below (i.e. what the next-highest bid was to our winning one) from a high-bidding campaign shows a pretty clear pattern: if you start at noon, the dollar cost for a thousand impressions (the CPM) generally drops until precisely midnight, then jumps something like 40%. It fluctuates a bit, hits a high in the morning and then drops down again until noon to start the cycle again. This pattern repeats day after day:

Notice the discontinuous jumps every midnight

So what’s with that midnight jump? This is supposed to be an efficient marketplace, but are impressions a minute past midnight really worth 40% more than impressions 2 minutes earlier? Had we not had our initial experiences described above it might have taken us a long time to puzzle out! Our current working theory is in fact that there must be lots of bidders out there that behave just like our first one did: they have a fixed daily budget and they reset their counters at midnight. This would mean that starting at midnight, there’s a whole lot more demand for impressions, and thanks to supply and demand, the price spikes way up (alternately, odds are that at least one of the many bidders coming back online is set up to bid higher than the highest-bidder in the smaller pool of 11:59PM bidders). And as bidders run out of budget throughout the day, at different times, depending on their budget and how quickly they spend it, the price drops to reduced demand. It’s not obvious from the above graph but when you plot the averages over multiple days together there is a similar step at 3AM, when it’s midnight on the West Coast, where a lot of other developers live and work and code:

Average Price Per Minute of the Day For the Month of August

Now there might be some other explanations for this price jump. We’re in the Northeast and that impacts the composition of our bid-request stream, so maybe this is just a regional pattern (do other people see this also?) although the 3AM jump suggests otherwise. Midnight also happens to be when our bid-request volume starts its nightly dip, but that dip is gradual and isn’t enough to alone explain that huge spike. Maybe some major sites have some sort of reserve price that kicks in at midnight. Or maybe everyone else is way smarter than us and noticed that click-through rates are much higher a minute past midnight than they are a minute before. I have trouble believing those explanations, though, so if anyone reading this has any thoughts on this I’d love to hear about them by email or in the comments below. Whatever the reason, it looks like if you’re going to only be bidding for a few hours a day, don’t choose the time-period starting at midnight and ending at 10AM!

Closed-Loop Control to the Rescue

So let’s assume that this post gets a ton of coverage among the right circles and that this before/after-midnight arbitrage opportunity eventually disappears, or maybe you just want to actually spread your spending evenly throughout the day, how would you do it? As I mentioned above, we have the ability to specify a percentage of the bid-volume to bid on. The obvious way to use this capability is to see how far into the day (percentage-wise) we get with our daily budget and bid only on that percentage of the bid-request stream, thereby ensuring that we run out of budget just as midnight rolls around. In practice, though, bidding on a fixed percentage of all requests can and does result in a different amount spent each day, sometimes higher than the daily budget (if you remove the daily cap) and sometimes lower. Maybe some days you get a different volume of bid-requests, because your upstream provider is making changes. Maybe there’s an outage. Maybe the composition of your bid-request stream is changing such that all of a sudden the percentage of all bid-requests that’s in the geo-area you’re targeting goes way up (or down). Maybe your bidding logic doesn’t result in a consistent spend. All of these factors could amount to a pretty jagged spend profile:

Spend rate can change despite constant bid probability

So what’s the solution? For us it’s to look at this problem as a control problem and to move from open-loop control to closed-loop control. If you theorize about the appropriate bid probability, set it up and let things run for a week and look at the results later, that’s basically open-loop control. If every morning you look at what the spend was the day before and adjust the coming day’s bid probability accordingly, you’re doing a form of closed-loop control: you’re feeding back information from the output into the input. It’s pretty easy to automate (at least we thought so), and once that’s done, there’s basically no reason not to shorten the feedback loop from daily to every few minutes.

Our RTB client was designed from the ground up to facilitate this sort of thing so we were able to control spend to within 1-2% of target with the following simple control loop: every 2 minutes, set the bid probability for the subsequent 2 minutes to that which would have resulted in the ‘correct spend’ over the preceding 2 minutes (maybe with a bit of damping). So if for whatever reason we get 20% fewer bid requests, for example at night, the bid probability will rise so that the spend rate stays roughly constant. There are more sophisticated ways to do closed-loop control, like the PID controller which I hacked into my espresso machine, but this really basic system works pretty well:

This is actual data from around the time we activated our simple controller. Notice how Open Loop Control gives output that looks like the mountains of Mordor whereas Closed Loop Control makes it look like the fields of the Shire

Actual production data, smoothed a little bit to reveal underlying trend. Pace (blue, prob*1000) rises at night when volume drops; Spend Rate (green) stays close to target (red)

Like any good optimizer, the scheme described above does exactly what you ask it to: it tries to keep the error between ‘instantaneous’ spend rate and the target as close to zero as possible. If there’s a full-on outage where no bidding occurs for an hour, this system will not try to make up the spend, it will just try to get the spend rate back to where it should be according to the originally-specified target, once bidding resumes:

Spend rate after outage is too low to catch up (slope remains parallel to initial target)

In order to deal with outages, as well as over- or under-spending due to accumulated error or drift from the instantaneous control system’s imperfect responses, we have to dynamically adjust this spend-rate target as well. The way we’ve come up with to deal with this is to define the error function in terms of the original problem: continuously work to keep the spend rate at a level that will spend the remaining budget in the remaining time. This means that if there is an outage for an hour or a day, the spend rate for the rest of the campaign will be marginally higher to make up for it, without human intervention. This system also has the nice property of automatically stopping right on the dot when the remaining budget is zero:

Spend rate after outage increases to catch up (slope gets steeper as outage gets longer)

Now obviously you might not want to always have exactly the same spend rate all the time, in fact I almost guarantee that you don’t. A closed-loop control system can support pretty much whatever spending profile you like. You can also use one to manipulate other variables than the bid-probability (think: the actual bid price, or the targeting criteria, or whatever other scheme your quant brain can conjure up).

Finally, the astute reader will notice that the approach described above is only really useful in situations where there is more supply than your budget permits you to buy, and so you need to allocate your budget over time. The approach above won’t help at all if the optimal bid probability to meet your needs is up above 100%, that is to say that even when you bid on and win everything that matches your targeting criteria, you’re not able to spend your budget. That’s probably a topic for a different blog post.

Follow-up: I also posted this question on Quora, where I got a few responses.

Follow-up posts: This post is now Part 0 of a series I’m calling Peeking into the Black Box. Check out Part 1 and Part 2 of the series, where I show how to really take advantage of other bidders’ behaviour.

There doesn’t appear to be a good Wikipedia entry for RTB for me to link at the moment, when I want to blog about it so I’ll draft my own explanation here. (Edit: there is an entry now, but I like my characterization better!) Keep in mind while reading this that I’m looking at RTB as a software engineer with an interest in economics, rather than as an ad industry veteran!

What is RTB?

Real-Time Bidding (RTB) is a way for advertisers to pay publishers to show ad impressions on their websites. In this context, an ‘advertiser’ can be anyone with a product to sell: a car company, a small e-commerce website etc. A ‘publisher’ can be anyone with a website: a newspaper, a blogger, Facebook, Google etc. An ‘impression’ is a single instance of a banner ad being shown on a web page. The term RTB is usually not applied to other ad formats such as text ads ads, or ads alongside Google search results or Facebook news feeds, as different technology is used to buy and sell those other formats.

The word ‘bidding’ comes from the fact that the price of impressions is set via an auction mechanism, where the role of the auctioneer is played by an ‘ad exchange’ who mediates between advertisers and publishers. The term ‘real-time’ stems from the fact that advertisers bidding in the auction must specify the price they are willing to pay for each impression separately for each impression, and within a strict time-constraint: usually around one tenth of a second, so the auction can run as the web page on which the impression will be shown is loading.

How does RTB work on Technical Level?

In terms of the technical aspects of participating in the auctions as a bidder, a computer subscribes to a stream of many thousands of bid-requests per second from the exchange, each one representing an impression to be shown very shortly. For each one, within the hard-real-time constraints, the computer must respond by specifying if/how much it wants to bid, in units of millionth of a dollar (μ$). The cost of online advertising is usually discussed in terms of cost-per-mille (CPM) in dollars, so $1 CPM implies 1 million microdollars for 1 thousand impressions, or μ$1000 per impression. The bidding computer can make its decisions based on the various attributes contained within the bid-request: the URL, the position of the ad spot in the page, the user’s location and local time, the number of times this user has previously been shown a given ad, any additional data the bidder has on file or has purchased on that particular user, etc.

What do RTB Auctions Look Like?

From my reading on auction theory, here is how I would characterize what goes on in most RTB auctions:

• There exists a sequence of auctions to sell impressions, and the participants (bidders) are not always the same from auction to auction.
• The auctions are closed (aka sealed-bid) and single-round, so participants do not see each other’s bids. All bidders submit their bids once and the winner is determined.
• The auctions are second-price, so the winner is the person who bids the highest, but pays whatever the amount of the next-highest bid was. Incidentally, only the winner knows what the winning bid was, and they learn what the next-highest price was, also, because that’s what they paid.
• The auctions are single-unit, in that participants are bidding on single ad impressions at a time.
• The goods being auctioned are perishable: you use them immediately when you buy them and then they’re gone. They have no resale value.
• The goods being auctioned are slightly different from auction to auction: the history of the user who will be seeing the eventual ad the winner will show matters to each bidder, as does for example the time of day etc.
• The valuations are uncertain, in that each bidder only has an estimate of how much it is worth to them to win this auction, and usually not even a very good estimate.
• The valuations are independent, in that the value of winning the auction to each bidder doesn’t impact the value of winning the auction to any other bidder.
• The valuations are private, in that the value of winning the auction to each bidder is not known to the others.
• Not every publisher is willing to sell to every advertiser, and vice-versa, due to various brand, appropriateness and competitive concerns.
• Publishers will sometimes set a reserve price: a price below which they will not sell, so as not to enable advertisers to use RTB to undercut existing sales channels.

These characteristics make the RTB marketplace fairly different from, say, a stock exchange, where people buy and sell large pools of indistinguishable units of stock, whose value is intimately linked with the price at which it can later be resold, and in general no one cares who is buying or selling.

Why?

Participating in RTB auctions is fairly technically challenging, and good bidding strategies are not obvious, so there must be some perceived advantage over the alternative. In this case the alternative is a complicated web of relationships between publishers and advertisers, sometimes with ‘ad networks’ and other middlemen in between. The purpose of RTB is to set up a more efficient marketplace for advertising inventory. One where fine-grained control over bids allows advertisers to save money by purchasing a smaller quantity of better targetted ads, even if they pay more per impression. One where publishers are able to earn more from their web properties because they can more economically reach the advertisers who are willing to pay a high price per impression for the ads they do show. And one where the end-users who actually see the ads are happier because they see fewer cheap and irrelevant ads.

Now obviously, for buyers to spend less and sellers to earn more something has to give, and that’s where the word ‘efficient’ comes in. Right now there is a lot of opportunity cost and operations cost on both sides of the market and RTB is meant to reduce a lot of that. Whether or not any of the promise of RTB is actually materializing for any side of the market equation is still an open question, and it’s a rapidly-evolving area of the online advertising industry.

More Reading & Listening

Mike on Ads is a good blog by the CTO of AppNexus, which is a big company in this space. Here are a bunch of his posts on RTB. I’ve also enjoyed a few episodes of the Trader Talk podcast on RTB.

One of the things we do at Recoset is to use machine learning algorithms to optimize real-time bidding (RTB) policies for online display advertising. This means we train software models to predict, for example, the cost and the value of showing a given ad impression, and we then incorporate these prediction models into systems which make informed bidding decisions on behalf of our clients to show their ads to their potential customers. The basic process of training a model is pretty simple: you  show it a bunch of examples of whatever you want it to predict and apply whatever algorithm you’ve elected to use to set some internal parameters such that it’s likely to give you the right output for that example. A trained model is basically set of parameters for whatever algorithm you’re using. Once you’ve trained your model, you test it: show it a bunch of new examples, to see if the training actually succeeded at making it any good at outputting whatever it is you wanted it to. The testing step often results in some datasets which we can feed into our data visualization system.

Now the bidding policies we’ve developed depend upon multiple such trained models, as mentioned above. Let’s use the example of a policy P which depends on a value model V and a cost model C, so P(V,C). We won’t need to train this policy per se, but we will want to run some simulations on it to test how good we would expect it to be at meeting our clients’ advertising goals. We might come up with 2 different types of value models (say V1 and V2) and 2 different types of cost models (say C1 and C2) to train and test.  So we end up with 4 possible combinations of value and cost models and thus 4 different possible policies to test: P(V1, C1), P(V1, C2), P(V2, C1), P(V2, C2). Note that in practice, we will likely come up with many more than 2 variations on value and cost models, but I’ll use 2 of each to keep this example manageable. So how do we actually go about doing all this testing and training efficiently, both from the point of view of the computer and the computer’s operator (e.g. me)?

When you’re just training and testing models independently of each other, you can easily get by with a simple loop in a script of some kind: for each model, call train and test functions. When you have some second step that depends on combinations of models, like the situation described above, you can just add a second loop after the first one: for each valid combination, call test function. This is how we first built our training and testing systems, but then we decided to parallelize the process to take advantage of our 16-core development machine. While brainstorming how we might build something which could manage the sequence of parallel jobs that would need to run, we realized that we’d already been using such a system throughout the development of these models: make. More specifically: make with the -j option for ‘number of jobs to run simultaneously’. The make command basically tries to ensure that whatever file you’ve asked it to make exists and is up to date, first by recursively ensuring that all the files your file depends on also exist and are up to date, and then by executing some command you give it to create your file. Casting our training/testing problem in terms of commands which make can understand is fairly simple and gives rise to the diagram at the top of this page: rectangles represent files on disk and ovals represent the processes which create those files. The only change required to our existing code was that the train and test functions had to be broken out into separate executable commands which read and write to files.

Here's the actual whiteboard brainstorm that led to this design :)

Why not just parallelize our simple for-loops using some threading/multiprocessing primitives? Because using those kinds of primitives can be difficult and frustrating, while using make is comparatively easy and gives us a number of other benefits. Beyond the aforementioned -j option which gives us parallelism, the -k option makes it less painful for some steps to error out without having to build any specific error-handling logic: make just keeps going with whatever doesn’t depend on the failed targets. And when we fix the code that crashed and re-make, make can just pick up with those failed targets, because all the rest of the files on disk don’t need to be updated and essentially act as a reusable cache or checkpoint for our computation (unless they caused the crash, at which point you delete them and make rebuilds them and keeps going). Having named targets also makes it easy to ask make to just give us a subset of the test results: it automatically runs only what it needs to to generate those targets we ask for. When you invoke make, you have to tell it what to make, in terms of targets, for example ‘make all_test_results’ or ‘make trained_model_V1′ etc. Adding new steps, such as the ability to test multiple types of policies, for example, or a ‘deploy to production’ step, becomes fairly trivial. Finally, the decomposition of our code into separate executables was actually helpful in decoupling modules and encouraging a more maintainable architecture. And whenever we get around to buying/building a cluster (or spinning up a bunch of Amazon instances), we can just prefix our executable commands with whatever command causes them to be run on a remote machine!

So, armed with our diagram, and our excitement about the benefits that using make would bring us, we implemented a script which essentially steps through the following pseudo-code, given a single master configuration file that defines the models to be used:

1. for each command to be run (ovals in the diagram), generate a command-specific configuration file and write it to disk, unless an identical one is already there
   • Setup, training and testing configuration files for V and C models
   • Testing configuration files for P policies
2. for each target (rectangles in the diagram), write to the makefile the command to run and the targets it depends on, adding the configuration file to the list of dependencies
   • Setup targets for V and C models
   • Training and testing targets for each V and C models
   • Testing targets for policies P for each combination of V and C models
3. call make, passing through whatever command-line arguments were given (e.g. -j8 -k targetName)

Step 1 is helpful in allowing us to change a few values in the master configuration file, and having make only rebuild the parts of the dependency tree that are affected, given that make rebuilds a target only if any of its dependencies have more recent file modification dates. This means that, for example, I can choose to change some value in the master configuration file related to the training of C models, and when I call make again, it will not retrain/retest any V models. Doing this in effect adds dependencies from the various commands to certain parts of the master configuration file and lets make figure out what work actually needs to be done. And, let it be said again: it will do everything in parallel!

Unfortunately, I don’t have any code to put up on GitHub yet as the script we wrote and I described above is very specific to our model implementation, but I wrote this all up to share our experiences with this structure and how pleased we are with the final result!

At Recoset, working with data often means data visualization (or dataviz): making pretty pictures with data. This is usually more like making fully machine-generated images than carefully laying out “infographics” of the Information Is Beautiful school but I find they usually end up looking pretty good. There are lots of good tools for graphing data, like matplotlib or R or just plain old Excel-clone spreadsheets but what we use most often is Protovis, the Javascript library for generating SVG, coupled with CoffeeScript, which is a concise and expressive language that compiles down to Javascript. The appeal of this combination for me is that it’s a very data-centric, declarative way of writing code to generate graphics. If you read the code for some of the very pretty examples on the Protovis site, you’ll see that they basically all have the same structure: a JSON-friendly object (usually an array) with all of the data to be visualized, and some small amount of code that declares how each element in that data object should be drawn. To me, the relationship between data and Protovis code is very similar to the relationship between semantic HTML and CSS: a separation of content and presentation.

When we first started using Protovis to make pretty pictures, we’d do something similar to the Protovis examples: make an HTML file which contained a bit of markup and a script block with the Javascript code to call Protovis, plus a script tag in the header, which would load an external file which contained our JSON data, preceded by something like “this.data=”. This suited us just fine, because the data to be visualized was usually the output of some Python or NodeJS or C++ process, and writing JSON to disk is really easy from pretty much any language. The thing is, once you’ve written a bunch of similar dataviz code a few times, you look for a way to reduce the repetitive work, and so you pull your code out into an external file and you end up creating a whole bunch of very small HTML files to combine code and data. Say you’ve settled on a nice way of graphing a certain type of data, for example the output of a per-partition Click-Through-Rate vs Cost-Per-Click analysis script. You’ll probably want to view the same graph for various partitions: per site, per vertical, per day of week, per time of day, whatever. So you generate a data file per partition, and you use the same code file, applied to each data file to generate each graph. And automating this task is just what the dataviz system that François and I built does.

At its core, it’s just a very small PHP file which generates an page with script tags that include a data file on the one hand and a code file on the other, based on GET parameters. This allowed us to create a few reusable Javascript or CoffeeScript files which specified how to visualize certain types of data, and use this script to quickly pull up a page for a given combination of data and code to have the browser show us the resulting SVG. Of course we don’t want to type GET parameters into the URL bar, so we wrapped this up in a little web-app that draws a file-tree based on what’s on disk and lets you click on the file you want and loads up our core PHP file in an iframe. No sweat. But then things get interesting.

Because Protovis is so data-centric and declarative, each code file implicitly requires a certain format in its data: some scripts work on arrays of arrays, others on dictionaries. We decided to create a simple convention for how we name our files on disk so our web-app would know which data files went with which code files. For example, a file in the ‘data’ directory called ‘stats-daily.js’ could be visualized with a code file in the ‘viz’ directory called ‘stats.js’. And then at some point we decided that some data formats could be visualized in more than one way, so we extended the code-file naming scheme to enable mixing and matching, say, data files named ‘stats-daily.js’ or ‘stats-hourly.js’ with code files called ‘bargraph-stats.js’ or ‘scattergraph-stats.js’ (see campaign example below). Essentially for a given data-type X, we have a one-to-many relationship with data files X-F1.js, X-F2.php etc and a one to many relationship with code files C1-X.js, C2-X.coffee etc. You’ll notice from the file extensions in the examples just given that the data files don’t have to be static on disk, they can be dynamically-generated with PHP. The code files can also be written in either Javascript or, my favoured option, CoffeeScript.

Basic Tree UI

The next thing we noticed after having used this system for a while is that sometimes you want to compare or otherwise look at 2 datasets of the same format, side by side. So we added generalized support for “multi” visualizations, where you can check off which data files you want to pass to the multi-data-set visualization code.

Multi-selection UI

So now whenever someone at the office wants to visualize some data, it’s just a matter of having whatever code is generating this data output a JSON file to our ‘data’ directory with the appropriate name for the format, and our app will display links to visualize the file with whatever appropriate code files exist. And if the desired visualization doesn’t exist yet, it’s just a matter of creating an appropriately-named file in the ‘viz’ directory, which can then be reused to look at other data files in the same format. This makes for some nice collaborative workflows where we’re all working on trying to build models that do the same thing and we can compare results really easily (see classifier example below).

Code & Demo

The code for our app is available on GitHub at https://github.com/recoset/visualize and a demo install of this code is available at http://visualize.recoset.com/ where you can see some sample data-sets and play with the UI. Google Chrome is the recommended browser for these visualizations as they’re pretty memory and CPU intensive so Firefox has trouble with them, and they’re SVG-based so IE has some trouble with that!

Classifier Example

If you go look at our demo, in the ‘classifier’ folder, you’ll see 3 data sets. Each data set is the result of training and testing a different type of classifier to predict a certain type of conversion. The data set is an array of objects, each of which contains the results of running the classifier at a specific probability threshold. We can use this data to plot Receiver Operating Characteristic (ROC) curves, as well as Precision-Recall (PVR) curves and Lift curves. In the screenshot below, you can see the ‘multi’ option in action, plotting the results of our Boosted Bagged Decision Trees, Generalized Linear Model and Stacked Denoising Autoencoders against each other (they seem to perform about the same for this task).

Precision-Recall Curves

Campaign Example

In the ‘campaign’ folder of our demo, you can see 2 datasets generated by our campaign-analysis system (note: this data has been obfuscated, and should not be interpreted as actual performance data!). The ’50 top hosts’ data is a good example of how we can visualize the same dataset in 4 different ways. In this case, the dataset is the performance of the 50 top hosts where we bought impressions over the course of this campaign. We can look at Click-Through Rate or Cost Per Click by host, (including confidence intervals!) or we can look at the relationship between these two quantities in a scatterplot. We can also look at a pie-chart of where we bought impressions. Same data, 4 views.

Click-Through Rate with Confidence Intervals

Cost Per Click with Confidence Intervals

CTR vs CPM scatterplot with CPC iso-curves

The second dataset in this folder allows us to draw what we call a campaign stream-graph. This is a visual representation of the progression of a campaign over time (day by day in this case, although we look at hourly versions as well). This graph shows various quantities of interest for each day of the campaign: impressions, total spend, CPC, CTR, CPM, site-mix etc. I encourage anyone interested in this visualization to go to our demo page and mouse over the various pieces of the graph.

Campaign Stream-graph

And now these great systems all play together really smoothly, thanks to Reeder, and to the Readability-Kindle integration. Reeder is a really smooth and easy-to-use Mac app which lets me plow through my Google Reader feeds and easily add content to Readability. And my Readability list is now synced to my Kindle just like a magazine subscription. So longreads posts a new article, which shows up in Reeder, I add it to Readability for later, and I can read it on my Kindle in the park!

This is a screenshot of what I pull up on my iPhone every morning now after its alarm clock wakes me up. That’s right, it’s an interface to turn on my espresso machine so that it will warm up to a specific temperature by the time I’m done snoozing! I can even look at a real-time plot of the temperature to confirm that it’s holding where it should be and doesn’t need to bumped up or down a degree.

More screenshots:

Silvia iPhone app icon (second from the bottom on the left)

Silvia iPhone graph

Why?

As outlined in this previous post where I laid out my plan, I wanted the extra precision of digital temperature control in my Silvia, but I didn’t want to slap a PID box on the side of it with an LED readout of the temperature. So I decided to build my own controller, and I opted to add Wifi to it for a web-based interface, which also opened up the possibility of turning it on remotely from bed, so it would be all warmed up by the time I was ready for my morning shot of espresso. Additionally, the controller automatically shuts off 1 hour after first being activated, so I essentially leave my Silvia powered up all the time, and the boiler only turns on when I ask it to.

How?

For the temperature control, I contacted the guy behind PIDKits.com and asked for everything except a PID controller, and he obliged, and also sold me a pre-assembled TC4v1 Arduino shield to interface with both the thermocouple (to read the boiler temperature) and the solid-state relay (to control the temperature). So with a bit of assembly, it was pretty easy to essentially bypass the built-in themostat with a digital one running on my Arduino. I played around with the P, I and D parameters on the controller, and I was able to get the temperature to be stable within ±1°C; much better than the ±15°C that the stock thermostat can achieve!

Next I ordered a CuHead Wifi shield for my Arduino and built a very simple REST-like interface using WiServer, which returns JSON, to run on the controller itself. This essentially turns the controller into a web service, which is in turned controller by a very simple PHP-based web app that runs on my iMac . This app contains all the right magical HTML incantations to look like a slick app on an iPhone/iPad, and can also be used on my iMac directly and even my new Kindle 3! (Update Jan 2012: I now have an HTML5 cached app on the phone: no iMac required after the first load!)

The whole thing is right now tucked away on the shelf beneath my Silvia, totally invisible to the casual eye, but I’ll probably put it in a little box or something just to hide exposed wires. (Update Jan 2012: in order to keep the connection stable I actually ended up hanging the controller on the wall hidden behind the Silvia.)

Thoughts

So I didn’t quite implement everything in the original plan, and I haven’t yet been able to run the entire app directly on the controller, nor have I been able to set up WebSockets through the Wifi shield, but overall I managed to accomplish the original design goals with a minimum of fuss, and hey, I use this thing every morning to make great espresso, so I’m a happy camper!

Links

• Arduino sketch on GitHub
• web-based “iPhone app” code on GitHub
• other posts about this

Thanks

• Jim Galt from PIDKits.com for all his advice
• The authors and contributors to the following open-source projects for making this type of hacking so much hassle-free fun:
  • the TC4 libraries
  • the WiServer library
  • SmoothieCharts
  • the Arduino toolchain

At Recoset we tend to worship, like Etsy (and AppNexus!), at the Church of Graphs. We’ve even started using Statsd, the system they’ve released to collect stats and relay them to Carbon for display in Graphite. And by display, I mean display on a dashboard visible to the entire dev team at the office, as seen above!

Statsd is a very simple system to which you can send UDP messages about various stats you want to track, which it then aggregates and passes along to Carbon, which stores them in Whisper, Graphite’s back-end data store. That’s a lot of moving parts but it works very well. Sending stats to statsd is extremely easy from any language (we do it from Javascript and C++) and carries low overhead, which is key for the type of work we do.

Graphite Screenshot

Graphite is basically a Django web app with a few different fancy front-ends to their “/render” URL, which returns lovely graphs depending on the query string, like you see above. There’s the ‘composer’ GUI interface, which is a point-and-click graph builder, as well as a web-based command line interface which can be scripted to generate lots of graphs quickly. If you tack on “rawData=true” to the query-string you pass to what they call the URL API, you get what you’d expect: the raw data that would have been used to generate a graph had that parameter not been set.

Now our dashboard doesn’t just show Graphite graphs, it cycles through multiple Firefox tabs (using the Tab Slideshow plugin) one of which is Opsview, which is a web front-end to the monitoring tool Nagios. We use Nagios to monitor a variety of systems, and to notify us if something goes wrong. Here’s a screenshot of Opsview telling us Nagios found nothing wrong and everything is great:

Opsview Screenshot

You can probably see where this is going: since we’re already shuttling stats to Graphite, and we want to use Nagios for alarming, and Graphite has this rawData mode… I built a generic little Nagios plugin called check_graphite which can be used to create Nagios service-checks so that it can monitor stats in Graphite and fire off alarms if needed. This was made pretty trivial by the excellent Nagaconda python module, but the end result is pretty powerful. We can now very easily set Nagios alarms on any stat we send to Graphite through Statsd, just by creating a service-check that contains the right query-string.

The check_graphite code is available on github under an MIT license.

Want to know more about what Recoset does or how our algorithms work? You should check out some other Recoset-related posts, such as “RTB Pacing: is everyone doing it wrong?” or the “Peeking into the Black Box” series.

(Update Aug 2011: for some of our most frequent stats we now bypass statsd and instead aggregate counters at their point of origin to send directly to Carbon, which is Graphite’s back-end. This cuts down on UDP messages and CPU usage considerably when sending tens of thousands of messages per second from one process through statsd :)

So I finally got around to working on my Silvia Mod Plan, getting all the way to Step 5! The video above is a demo of the setup I have to show a real-time graph on my iPad of the boiler temperature in the Silvia.

Having installed the thermocouple in the Silvia and played with my TC4 shield, my initial plan was to use the Arduino to transmit data to my iMac using XBee as a wireless serial link, where I would run a NodeJS process which would read data from the USB port and which would communicate with the iPad via a WebSocket over Wifi (phew, mouthful!). Ideally the Arduino would speak Wifi but in the meantime I figured I’d play with this setup. I chose NodeJS because it seemed really easy to set up WebSockets using socket.io, and that seemed like a good way to feed data to Smoothie Charts for real-time graphing. I rewrote the code in CoffeeScript, because it’s the best way to write NodeJS code IMO (a discovery I made after writing the first version of this code 4 months ago) and because it’s so fitting for this project!

The first snag I ran into was the fact that there doesn’t seem to be a good way to read from a USB port from Node, and it felt too much like my day-job to write an add-on in C++ to do it, so I had to work around. My solution was to run a separate Python process which would read data from the USB port and would POST to the NodeJS web server whenever the Arduino sent a temperature reading.

The second snag I ran into was the fact that the XBee didn’t seem to be able to transmit all the way to my iMac, so I hung the XBee from my laptop, located halfway between the iMac and the Silvia and ran the Python script there, POSTing over Wifi to the iMac. Then I smartened up and installed the NodeJS process on my laptop. So really, this system is just a convoluted way of doing what could have been done using a USB cable to graph data on my laptop, but bouncing signals around over multiple protocols, machines and wireless technologies is so much more fun!

The code is, of course, up on Github: https://github.com/nicolaskruchten/coffeegraph