Age gaps and Birth order: Failed reproduction of results


I at­tempted to re­pro­duce Scott’s anal­y­sis of Birth or­der effect vs Age gap. I found that:

1. There ap­peared to be an er­ror in graphs 2 & 3 where peo­ple with one sibling were counted when they shouldn’t have been (graph 2) or were counted twice (graph 3)

2. Com­par­ing old­est chil­dren to youngest chil­dren causes a sys­tem­atic bias in the re­sults. This can be pre­vented by com­par­ing old­est chil­dren to 2nd old­est children

3. I was un­able to re­pro­duce Scott’s re­sult on peo­ple re­port­ing 0 year age gap – I get a non-sig­nifi­cant 58% older siblings com­pared to Scott’s 70%. I was un­able to dis­cover the cause of the differ­ence.

I have re­analysed the data based on points 1 & 2 in a sep­a­rate post.

Pre­vi­ously in Birth or­der effect

In the 2018 Slate Star Codex sur­vey Scott asked some ques­tions about what or­der in the fam­ily re­spon­dents were born. He found that el­dest chil­dren were mas­sively over­rep­re­sented.

Fol­low­ing on, his­tor­i­cal math­e­mat­i­ci­ans and No­bel win­ning physi­cists were found to ex­hibit the same prop­erty.

In the 2019 SSC sur­vey Scott in­cluded ques­tions about age gaps be­tween re­spon­dents and their ad­ja­cent siblings. He analysed the re­sults, find­ing that:

This study found an am­bigu­ous and grad­ual de­cline [in Birth or­der effect] from one to seven years [Age gap be­tween siblings], but also a much big­ger cliff from seven to eight years.

Failed re­pro­duc­tion of Scott’s graphs

I had origi­nally in­tended to analyse the data to see if I could draw any fur­ther con­clu­sions. How­ever, when run­ning the anal­y­sis I found that I was un­able to re­pro­duce Scott’s re­sults.

Scott in­cludes 3 graphs.

The first – com­par­ing % of sam­ple old­est child vs age gap for peo­ple with 1 sibling – I was able re­pro­duce al­most ex­actly (Scott also has ac­cess to re­spon­dents’ data who asked not to be in­cluded in the pub­lic data so we aren’t ex­actly the same. There may be other differ­ences too but these are small).

The sec­ond – com­par­ing how many old­est vs youngest chil­dren there are in the sam­ple for peo­ple with more than 1 sibling – I was un­able to re­pro­duce. Ac­tu­ally, I was able to re­pro­duce the graph but only if I also in­cluded peo­ple with 1 sibling.

This is ac­tu­ally what the third graph was sup­posed to show, but it looks like the third graph dou­ble counts the peo­ple with 1 sibling.

The graphs are be­low, with Scott’s on the left and my re­pro­duc­tions on the right. Note the similar­ity be­tween Scott’s graph 3 and my graph 2.

(My ver­sion of graph 2 has a differ­ent y-axis to all of the other graphs as the range is larger)

As an in­tu­itive way of see­ing that there is some­thing wrong – Scott’s third graph lists 7,613 sam­ples in­cluded. There are only 8,171 peo­ple in the whole sur­vey but the third graph should be rul­ing out any only chil­dren and any chil­dren be­tween first and last in their fam­ily. It seems that there should be more than 558 peo­ple in that group (ac­tu­ally there are 767 only chil­dren, not even count­ing any mid­dle chil­dren).

(Scott’s re­sults were dou­ble checked by Tum­blr user athenae­galea but this only in­volved look­ing at the data for the first graph which I also agree with).

Cor­rect­ing this mis­take is im­por­tant as the 7 year age gap drop in birth effect is pred­i­cated on graphs 1 & 2 both show­ing such a drop. In my re­pro­duc­tion there is no such drop in graph 2 which sug­gests the 7 year age gap sud­den drop may not be a thing af­ter all.

Prob­lem with com­par­ing old­est and youngest children

Below I have plot­ted the data from graphs 1 & 2 on the same axis (switch­ing to line graphs to make trends a bit eas­ier to see). I have changed the y-axis to be a ra­tio of old­est:youngest in­stead of a per­centage to high­light the strangeness of the re­sults.

The 1 sibling data still show a rel­a­tively con­stant birth effect across differ­ent age gaps un­til 7 years gap. 8 years and above then shows a drop in Birth or­der effect size.

The >1 sibling data show a very differ­ent trend. At 1 year age gap there are over 7x more first­borns than last­borns. This de­creases rapidly as age gap in­creases. Above 9 years age gap there are ac­tu­ally more youngest chil­dren than old­est (al­though the sam­ple size is rel­a­tively small here).

This seems odd – the two data sets should be re­flect­ing roughly the same pro­cess but with differ­ent fam­ily sizes – the graphs should be similar, or at least closer than this! Does hav­ing ad­di­tional siblings qual­i­ta­tively change how age gap mod­ifies the Birth or­der effect?

Above we are com­par­ing old­est siblings with youngest siblings and look­ing at the rel­a­tive age gaps. In do­ing so we are im­plic­itly as­sum­ing that the age gaps be­tween 1st and 2nd chil­dren should be statis­ti­cally similar to the age gaps be­tween penul­ti­mate and last chil­dren in the gen­eral pop­u­la­tion.

This does not match with my ex­pe­rience. In fam­i­lies with >2 chil­dren that I know, age gaps be­tween later chil­dren tend to be larger than be­tween ear­lier chil­dren.

Check­ing this in the data, I looked at peo­ple who are nei­ther first nor last chil­dren and com­pared the age gaps to their next older sibling and next younger sibling. On av­er­age the age gap to their younger sibling was 0.55 years longer than the gap to their older sibling.

This effect would ex­plain the in­cred­ibly high birth or­der effect with 1 year age gap seen in my graph 2. If many old­est and few youngest chil­dren in the gen­eral pop­u­la­tion have 1 year gap to their neigh­bour­ing sibling and we add this to the SSC Birth or­der effect then the over­all effect in the SSC sam­ple will be huge.

It would also ex­plain how the birth or­der effect in graph 2 ap­pears to de­crease dra­mat­i­cally for fam­i­lies with large age gaps – in the pop­u­la­tion as a whole, more youngest than old­est chil­dren in the over­all pop­u­la­tion will have >5 year gap to their neigh­bour­ing sibling. This over­all pop­u­la­tion effect can­cels out some (or all for >9 year gap) of the Birth or­der effect from the SSC pop­u­la­tion.

I can’t think of a way to quan­tify and/​or can­cel out this effect whilst com­par­ing old­est to youngest chil­dren but for­tu­nately we don’t have to.

In­stead of com­par­ing old­est and youngest chil­dren, we could com­pare first and sec­ond chil­dren. If we only look at first and sec­ond chil­dren and com­pare age gaps down­wards and up­wards re­spec­tively then we should be look­ing at the same un­der­ly­ing dis­tri­bu­tion of age gaps in the gen­eral pop­u­la­tion.

One ad­van­tage of com­par­ing old­est to youngest siblings is that this rep­re­sents the largest Birth or­der effect (see Scott’s 2018 anal­y­sis). How­ever, as most of the effect hap­pens be­tween first and sec­ond siblings, the effect should still be large enough to de­tect us­ing these sam­ples.

Re­do­ing my pre­vi­ous graph based on first and sec­ond chil­dren rather than first and last gives some­thing which makes a lot more sense—there isn’t much differ­ence be­tween 1 sibling and >1 sibling. The >1 sibling data set doesn’t have the sud­den drop af­ter 7-years but does sug­gest a slight down­wards trend around the same time.

Re­cre­at­ing Scott’s graph 3 (i.e. in­clud­ing all fam­ily sizes) gives a drop in birth or­der effect at 7 years but not as low as with just the 1 sibling data – ~59% old­est chil­dren vs 2nd old­est, com­pared to ~54%.

Failed re­pro­duc­tion of birth or­der effect with 0 year age gap

I also failed to re­pro­duce Scott’s find­ing re­gard­ing peo­ple re­port­ing a zero year age gap. He finds that:

Weirdly, among peo­ple who re­ported a zero-year age gap, 70% are older siblings.

but I was un­able to pro­duce a re­sult like this.

First I re­moved the re­spon­dents who re­ported 0 year age gaps in a di­rec­tion in which they re­ported 0 siblings. This re­moved over half of the re­ported 0 year age gaps.

Of those re­main­ing, I get a non-sig­nifi­cant 58% older siblings (53 vs 39). There were also 3 re­spon­dents who in­di­cated that they were in the mid­dle of a mul­ti­ple birth.

In this case I’m not sure how I get such differ­ent re­sults from Scott. Even if I don’t re­move the re­sponses as I de­tailed above I don’t get any­where near Scott re­sults (I ac­tu­ally get ~60% younger sibling – with so many old­est chil­dren there are more op­por­tu­ni­ties to put a 0 years in the up­wards di­rec­tion where it should have been left blank).

So I’m very con­fused about why I’m get­ting such differ­ent re­sults.

In my next post I re­analyse the data based on 1st and 2nd old­est chil­dren.