Using online ads for political advocacy
I once heard Slate commentator Jamelle Bouie say, on Twitter, that you shouldn't underestimate older black women as footsoldiers for the Democratic party. However, it wasn't until I started running Facebook ads to promote a petition that I started to get this on a gut level.
I didn't set out to study this. But by default, Facebook's ads dashboard displays the age and gender of people clicking on your ads (Facebook doesn't do this for race). And the difference on the bar graph was big enough to be hard to miss. I ran a simple statistical test on the gender component, and the odds the difference was due to chance were low, but not quite low enough for me to feel sure I wasn't fooling myself. Once I'd have enough data that the odds of a spurious result were less than half a percent, though, I stopped advertising the petition to men entirely.
I feel a bit weird about this. I'm a man, and it's not like no men were signing the petition, so by excluding men I knew I'd be missing out on potential supporters. But the goal wasn't to get every single person who might be persuaded to sign the petition to sign it—ultimately, the goal is always to get enough people to sign it to have an impact. Given a limited budget, I think focusing effects wherever you get the most bang for your buck is the right choice.
The age-based difference looked pretty big to the naked eye too, just from the bar graph, but I worried if I approached too many questions that way, I'd end up fooling myself a lot. So I set up an actual experiment: one "ad set" targeted to women 18-40, one targeting women 41+, and this time promising myself I'd let the experiment run for a whole week, no matter how clear initial results looked. At the end of the experiment, once again the evidence was clear: older women were more likely to sign the petition.
I've also done tests on targeting different states. Because I'm focusing on opposing Trump's appointments, and Trump needs the Senate to confirm his appointments, I've focused on states that have at least one Republican senator who was publicly refusing to endorse Trump on November 8th. In this case, I haven't (yet) been able to find differences big enough to justify doing what I would otherwise do, which is focus on states that (1) are small, so for any given percentage of the population, it's easier to get that many signatures (2) have two Republican senators rather than one.
This probably means targeting Alaska and Nebraska, though I haven't yet ruled out giving special attention to Utah as well. Specifically, my estimated cost-per-signature is $1.69 for Alaska, $1.40 for Nebraska, and $1.14 for Utah. The margin of error for these numbers is about plus or minus 20%, so it's hard to tell if the apparent difference between Nebraska and Utah is real, and more importantly whether it's big enough to make up for Utah's bigger population.
For those who aren't up on statistics, one of the things that makes this project tricky is that the more questions you ask a dataset, the greater the odds are you'll get a nonsense result by chance. The way you counteract this is demanding stricter tests of statistical significance, which usually means needing more data. In other words, the more questions you want to ask, the more data you'll need, if you want to do it right. Since this project is in its early stages, I don't have a ton of data. Yet.
Questions I'm hoping to be able to answer in the future include: to what extent do the ads suffer from diminishing returns? Does the day of the week I run them on matter? Do any of the grab bag of things (other than gender) that I'm currently using as proxies for "probably doesn't like Donald Trump" actually work? If so, which ones? Do any work better than gender? (A "yes" to the last one would be great, then I could stop feeling weird about ignoring men.)
Right now, when someone signs the petition, if they fill out their e-mail address they'll get an automated e-mail encouraging them to actually call their US Senators opposing Jeff Sessions' nomination. It's hard to tell if this e-mail works, though. I've considered sending out a survey, but then I'd have to contend with the non-response problem. That's why if the Defend Democracy PAC raises enough money, I hope to use a service like Mobile Commons to make it maximally easy for people to make that call, and then track what actually works for getting people to do so.
If you think this sounds worth doing, please donate now. Every bit helps.