How does gender affect X?

An interesting question from Allen Riddell popped into my inbox yesterday. I thought the discussion would be interesting to some readers of the blog. 

I've got a case where I think an IV might be useful. Do you have a tutorial on this ready yet? (I recall you mentioning you were writing one.) Or perhaps there's a book/journal article you'd recommend?
I'll sketch out the case I'm working with in the event that it's not a case where IVs would be useful:
I'm trying to see if there's gender bias in the books (fiction) selected for review in a magazine. I've got a population of books and only some of them are selected for review.
The problem is that the genre of the book (e.g., thriller, romance) is correlated with gender AND the magazine is open about only reviewing certain kinds of books. So it's hard to say whether or not a bias against women-authored books isn't just a bias against genres in which they tend to predominate. 
I have a continuous variable which measures genre -- so right now it's a logistic regression with reviewed ~ gender + genre. Perhaps an IV would help?

My response:  

Hey Allen, Good to hear from you. I've also included Adam and Cory. Allen does analysis on text at Indiana U, and wrote pystan. Adam does causal inference thinking at Buzzfeed, and recently wrote a package on graphical causal inference. Cory is an economics PhD student at MIT and has very strong causality chops.
Just to be clear, you're looking for the causal path from an author's gender to their being reviewed, but you're saying that gender has an influence on genre (which will also influence review probability).
If there is a causal path from gender to genre here, then genre is a post-treatment control that will bias your effect estimate. That is, some of the causal effect of being female on review probability is being soaked up by their choice of genre. If you randomly vary their gender at birth, they _don't_ write the same book; they write a different genre which has a different probability of being reviewed (but this choice is caused by their gender). So if you're genuinely willing to estimate the causal effect of gender on being reviewed, you would drop the genre covariate.
But that's not the interesting question. If I read between the lines, your research question is more "if I randomly change the gender of a given book's author, how does it's probability of being reviewed vary". If so, you need to do more work than the current set-up.
I'm also not sure that an IV approach would work here, simply because a good instrument should result in exogenous variation in the treatment (gender). Unless your population of authors come from gender-selectively aborted populations in Chinese villages, or publishers with gender quotas, it'll be hard to argue for a good IV.
One approach would be to go from actual gender to perceptions of gender (which might be able to be exogenously varied by, say, names). I can't recall where, but I recall hearing about someone using initialization of first-middle names to help inform a perception of gender. Ambiguous names would also be. The idea being that initialized first-middle names (J.K. Rowling etc) are ambiguously gendered. The assumption here is that the reviewer "perceives" gender after the book has been written. 

[I go on to recommend a model, but that's not so interesting]

Next, Cory: 

Jim basically said what I was going to! Gender is a very weird case because it IS randomly assigned at birth (perhaps along with a few other X/Y chromosomal genes) so in some sense *all* regressions of something on gender are valid in reduced form!
We now run into two problems. First, since this was an RCT from conception, literally anything else that happens to the person is a pathway through which gender can matter. If you could be SURE that genre was the ONLY other factor that mattered you'd be fine with controlling for it, but that's a tough claim to make. The risk, as Jim says, is that people select into genres based on characteristics we don't observe (not gender). And then controlling for genre is just picking up the effects of those characteristics.
If you want to just account for genre to see genre-adjusted gaps, you should find an instrument for *genre* rather than gender.
Jim's idea of manipulating *perceived gender* is a good one & many studies have used this approach to good effect! The best approach is randomly changing the name on the manuscript to suggest a man vs a woman. Or making the process name blind. 

Then Adam: 

Hi All!
Hard problem! I think there's a problem with conditioning on genre to remove the effect of genre on acceptance, and studying the remainder of the total effect. It's even harder if you want to work with gender. I analyze sex first, then say how they'll be different.
First, the question, to make sure I have this right: It sounds like you're looking for "After accounting for acceptance differences due to genre, is there a sex bias in acceptance rates?", and a secondary question "Is there a sex difference in acceptance rates".
(1) IV isn't great, since, as the others mentioned, sex is randomly assigned already. It'd be hard to find an instrument for it! You'd need to identify an effectively randomly controlled cause of sex. Fortunately, you don't need to -- it's already randomly controlled! To be complete, the basic structure that lets you do IV looks something like (it turns out this picture is a little too strict -- the Z --> X arrow could also be from confounding between Z and X, i.e. Z <--> X, and that's perfectly fine!)

where Z is the instrument, X is the cause, Y is the effect, and U is a set of (possibly unobserved) confounding variables. We assume Z doesn't cause Y except possibly through X. More precisely, we assume (1) Z is not independent of Y, and (2) If we remove the causes of X, Z and Y become independent. Having made these assumptions, we can identify a conditional effect: the effect of X on Y for the subset of the population who is induced into the treatment by Z.
(2) If you want to control for genre to remove the intermediate effect S --> G --> A to focus on the total effect (minus that path), you have to make some additional assumptions. I believe the most general graph we can write, assuming RCT for sex, S, writing genre as G, confounders as U, and acceptance as A, looks like this

If we control for G, we break the causal path S --> G --> A, but introduce spurious correlation by conditioning on the collider at G along the path S --> G <--U --> A. To avoid this, you'd have to be willing to assume no confounding between genre and acceptance! That doesn't seem like something you could assume without a _really_ good reason. Note the edge from S --> U isn't required for this problem to happen.
Bottom line:It looks like the only identifiable effect from this graph is the effect of S on any measured variable.
Introducing gender, which is adopted by the individual at some point, removes randomized control for S (unless we assume gender is 100% biological). Then, you need additional confounding pointing to all variables, and no effect is identifiable.

Allen responds: 

This is all really helpful. Thanks!
Let me add a small wrinkle:
Could matching or some similar approach help somewhat? For example,books are also associated with publishers, and publishers tend to preferto offer book contracts to writers likely to be commercially successful(i.e., they care about different things than the reviewing magazine).Moreover, we have, typically, the full text of the books so we canestimate the "quality" of the book to an appreciable extent.
Re: sex vs. gender I think it's a safe simplifying assumption thatgender identity is randomly assigned. And we can restrict the analysisto authors who are writing under a name which is aligned with theirvisible gender. (Most books feature author photos.)
Also, I think I might be working with subtly different assumptions. Thisis the scenario I'm working with:
1. Magazine reviews books which are classified as genre 1 ("thriller")and not genre 2 ("romance"). (I know I said genre was a continuousvariable, but this is barely true; signs of genre are as conspicuous asthey get)2. Women and men write books in these genres at very different rates.The worry is that men and women are writing in these genres differsystematically.
So, in general, you can predict with very high probability if a book isthe kind of book which the magazine will *not* review under anycircumstances. Adam, does this address what you were saying aboutassuming "no confounding between genre and acceptance".

Then Adam: 

If you're satisfied that you've controlled enough covariates to control for all of the confounders of genre and acceptance, then they do help! If you're able to block the path which is opened by conditioning on G, S --> G <--U --> A, by conditioning on U as well, that's a perfectly legitmate thing to do. Then you've removed the spurious dependence (collider bias) for the remainder of the total effect of gender on acceptance (minus the path through genre).
Generally, a claim that you've accounted for all confounders is pretty dubious. After controlling, how can you be sure the remaining statistical dependence between genre and acceptance isn't due to yet more unobserved variables?