{"id":501,"date":"2018-07-10T16:05:03","date_gmt":"2018-07-10T21:05:03","guid":{"rendered":"http:\/\/web.education.wisc.edu\/nwhillman\/?p=501"},"modified":"2018-07-10T16:05:03","modified_gmt":"2018-07-10T21:05:03","slug":"using-difference-in-differences-in-higher-education-research","status":"publish","type":"post","link":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/2018\/07\/10\/using-difference-in-differences-in-higher-education-research\/","title":{"rendered":"Using difference-in-differences in higher education research"},"content":{"rendered":"

Difference-in-differences is gaining popularity in higher education policy research and for good reason. Under certain conditions<\/a>, it can help us evaluate<\/a> the effectiveness of policy changes.<\/p>\n

The basic idea is that two groups were following similar trend lines for a period of time. But eventually, one group gets exposed to a new policy (the treatment) while the other group does not (the comparison). This change essentially splits time in two, where the treatment group’s exposure to the policy puts them on a new trend line. Had the policy never been adopted, the two groups would have continued on similar paths.<\/p>\n

I made a data file<\/a>\u00a0and show the steps I took to conduct this analysis in Stata.<\/p>\n

Step 1: Generate treatment variable<\/strong><\/p>\n

Let’s say College A is exposed to the policy change (treatment) and College B is not (comparison). We just need a simple dummy variable to categorize these two groups. Let’s call it “treat,” where College A gets a value of 1 and College B gets a value of 0.<\/p>\n

\"\"<\/a><\/p>\n

Step 2: Generate “post” policy variable<\/strong><\/p>\n

In the previous step, we didn’t say when<\/em> the policy change occurred, so we need to do that now. Let’s say it began in 2015, meaning all years from that point forward are “post-policy” while those prior are “pre-policy.”<\/p>\n

\"\"<\/a><\/p>\n

Step 3: Examine trends for the two groups<\/strong><\/p>\n

Now that we’ve identified the treatment\/control and the pre\/post periods, we can put it all together in a simple graph. I like to use the user-written “lgraph” command (use “ssc install lgraph, replace” to get it).<\/p>\n

We see here the two groups were following similar trends prior to the 2015 policy change, and then the treatment group started to increase at a higher rate while the comparison group did not.<\/p>\n

\"\"<\/a><\/p>\n

Step 4: Difference-in-differences means table<\/strong><\/p>\n

The visual inspection looks like there’s probably a policy effect, but it’s hard to tell the magnitude. To get at that, we need to measure the difference<\/em> in groups means before and after the policy:<\/p>\n

\"\"<\/a><\/p>\n

Below is a simple table calculating the difference between the two groups before (-10) and after (5) the policy. It then calculates the difference before and after within each group (20 and 35, respectively).<\/p>\n\n\n\n\n\n\n
<\/td>\nPre<\/td>\nPost<\/td>\nDifference<\/td>\n<\/tr>\n
Comparison<\/td>\n517.5<\/td>\n537.5<\/td>\n20<\/td>\n<\/tr>\n
Treatment<\/td>\n507.5<\/td>\n542.5<\/td>\n35<\/td>\n<\/tr>\n
Difference<\/td>\n-10<\/td>\n5<\/td>\n15<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The comparison group increased by 20 units after the policy, while the treatment increased by 35. Similarly, the treatment group was 10 units lower than the comparison group prior to the policy, but was ahead by 5 units after. When we calculate the difference in the group differences, we get 15<\/strong>\u00a0(e.g., 35 minus 20, or 5 minus -10).<\/p>\n

Step 5: Difference-in-differences regression<\/strong><\/p>\n

We can run a regression on the data using the two variables created in Steps 1 and 2. The only trick is we need to interact those two variables (treat x post) to get our difference-in-differences estimate.<\/p>\n

\"\"<\/a><\/p>\n

Here, the “treat” dummy measures the treatment group’s pre-policy difference from the comparison group. And “post” is the comparison groups post-policy change. The interaction between the two variables (“treat x post”) is our average treatment effect of 15<\/strong>, the same number we saw in the previous step. The intercept is the comparison group’s pre-policy mean.<\/p>\n

In a regression framework, we can easily add covariates, use multiple comparison groups for robustness checks, and address issues that may arise with respect to standard errors. Doing so can help rule out plausible alternative explanations to the findings, assuming other important<\/a> considerations<\/a> are also met.<\/p>\n

My goal with this post was to break down the difference-in-differences approach to help make it a little more accessible and less intimidating to researchers\/policy analysts. I am still leaning a lot about the technique, so please consider these steps some illustrative tips to get oriented\/introduced.<\/p>\n

Stata replication code:<\/p>\n

\/\/ generate treatment variable (College A is the treated unit, College B is comparison)
\ngen treat = 1 if id==1
\nrecode treat (.=0)
\nlab def treat_lab 0 “comparison” 1 “treatment”
\nlab val treat treat_lab
\ntabstat y, by(treat) stat(n mean min max sd)<\/p>\n

\/\/ generate post-treatment period (2015 is start date)
\ngen post = 1 if year>=2015
\nrecode post (.=0)
\nlab def post_lab 0 “pre” 1 “post”
\nlab val post post_lab
\ntabstat y, by(post) stat(n mean min max sd)<\/p>\n

\/\/ descriptives of the two groups
\ntable year treat, c(mean y)
\nssc install lgraph, replace
\nlgraph y year, by(treat) stat(mean) xline(2015) ylab(, nogrid) scheme(s2mono)<\/p>\n

\/\/ did means table
\ntable treat post, c(mean y)<\/p>\n

\/\/ did regression
\nxtset id year
\nxtreg y i.treat i.post i.treat#i.post<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"

Difference-in-differences is gaining popularity in higher education policy research and for good reason. Under certain conditions, it can help us evaluate the effectiveness of policy changes. The basic idea is that two groups were following similar trend lines for a period of time. But eventually, one group gets exposed to a new policy (the treatment) […]<\/p>\n","protected":false},"author":22,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[1],"tags":[],"class_list":["post-501","post","type-post","status-publish","format-standard","hentry","category-community-college-research"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/posts\/501","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/comments?post=501"}],"version-history":[{"count":0,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/posts\/501\/revisions"}],"wp:attachment":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/media?parent=501"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/categories?post=501"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/tags?post=501"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}