This is an update on our pink bagel tests, which we first wrote about on this blog a week ago. To read more about the motivation and setup of the tests, check out our first pink bagel blog post here.
During the Test
We ran our campaign on Facebook's ad platform as discussed in our previous blog post. We started the campaigns on 2/24 and ran them until 3/4. Initially, we had all six possible ad creatives running in the same ad set, targeting EV owners in the Bay Area. However, we quickly discovered that Facebook's algorithm was unequally allocating spend to our ads. In other words, one of our creatives was getting nearly all of our money, and the other creatives were only rarely being served. To fix this problem, on Saturday, we decided to create six different identical ad sets, each one running one of the six creatives. This way, we could allocate spend manually to each ad so that they could get served with near equal frequency. Initially, we were spending $10/day on ads; after this change, we were spending $1.50/ad/day until 3/2, when we began spending $2/ad/day. This spend was sufficient for us to get statistically significant results.
The results for our actual Pink Bagel test were quite good. Recall that, for our pink bagel, we were comparing two different ad texts:
- Text 1: “Help enable EV owners to request a public charging station by checking out our prototype!”
- Text 2: “Frustrated with unavailable public charging stations? Check out our prototype!”
The goal was to understand how calls to action can affect ad performance; in this case, we were comparing an assertion to a question.
We found that text 1 performed better than text 2. Text 1 achieved a click through rate (CTR) of 0.98%, while text 2 had a CTR of 0.89%. Due to our sample size, these results were statistically significant (for details, check out this slide).
Unfortunately, our attempt to get feedback on our prototypes did not go as well as the Pink Bagel test. Using bit.ly click trackers, we found that many people did click to our prototype, but only a few went to the survey afterwards (for prototype 1, 29% of visitors went from the prototype to the survey. For prototype 2, about 35%, and for prototype 3, 40%). As further indication that this survey thing wasn't going to work, we only got two responses between the three surveys.
- People respond better to ad text that contains assertions than questions.
- However, getting people to engage with a landing page is hard, and getting them to go further than the landing page is even harder.
- If we want to get feedback from strangers on the internet, ads and surveys are not the way to go.
- Facebook's ad platform does not do what you expect it to do, so if you have specific ad spend targets for each ad, make sure to allocate that money manually.
Learnings from other teams
- A couple of teams did an interface pink bagel test instead of an ad pink bagel test, and had useful results. At least one team used Optimizely to do so, to good results.
- Many teams built landing pages manually and used things like custom click trackers, to good results.
- Some teams compared images in ads instead of text in ads and saw more dramatic results than we did.
- Many teams targeted the entire US instead of just the Bay Area, also to good results.
Where we go from here
At this point in our project, we are not at a point where we need to use ads to acquire users, so we do not have any immediate action items from these tests. These tests will likely shape how we run our ad campaigns in the future.
More immediately, however, we need to find better ways to get in front of electric vehicle users. In conjunction with these tests, we also posted our prototypes to EV forums, to little success. Therefore, we will have to resort to other means (perhaps via our liaisons?) to get in front of EV owners. We have had the most luck with simply going to EV charging stations and talking to people, so that is likely where we will look in the future when we need input from EV owners.
To see the full results from our experiment, check out our presentation here.