{"id":396,"date":"2017-07-21T14:16:25","date_gmt":"2017-07-21T19:16:25","guid":{"rendered":"http:\/\/web.education.wisc.edu\/nwhillman\/?p=396"},"modified":"2017-07-21T15:31:00","modified_gmt":"2017-07-21T20:31:00","slug":"why-we-need-comparison-groups-in-pbf-research","status":"publish","type":"post","link":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/2017\/07\/21\/why-we-need-comparison-groups-in-pbf-research\/","title":{"rendered":"Why we need comparison groups in PBF research"},"content":{"rendered":"

Tennessee began implementing performance-based funding in 2011 as part of its statewide effort to improve college completions. The figure below uses IPEDS completion data plotting the average number of bachelor’s degrees produced by each of the state’s 9 universities over time.<\/p>\n

\"\"<\/a><\/p>\n

One could evaluate the policy by comparing outcomes in the pre-treatment years against those in the post-treatment years. Doing so would result in a statistically significant “impact,” where 320 more students graduated after the policy was in effect.<\/p>\n\n\n<\/colgroup>\n\n\n\n
<\/td>\nPre<\/td>\nPost<\/td>\nDifference<\/td>\n<\/tr>\n
Tennessee<\/td>\n1940<\/td>\n2260<\/td>\n320<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

This Interrupted Times Series approach was recently used in a report concluding that 380 more students (Table 24<\/a>) graduated because of the policy. My IPEDS estimate and the one produced by the evaluation firm use different data, but are in the same ballpark.<\/p>\n

Anyway, simply showing a slope changed after a certain point in time is not strong enough evidence to make causal claims. In very limited conditions<\/a> can one make causal claims with this approach. But this is uncommon, making interrupted time series a pretty uncommon technique to see in policy research.<\/p>\n

A better approach would add a comparison group (a la Comparative Interrupted Time Series or its extensions). If one were to do that, then they would compare trends in Tennessee to other universities in the U.S. The graph below does that just for illustrative purposes:<\/p>\n

\"\"<\/a><\/p>\n

By adding a comparison group, we can see that the gains experienced in Tennessee were pretty much on par with trends occurring elsewhere in the U.S.:<\/p>\n\n\n<\/colgroup>\n\n\n\n\n\n
<\/td>\nPre<\/td>\nPost<\/td>\nDifference<\/td>\n<\/tr>\n
Tennessee<\/td>\n1940<\/td>\n2260<\/td>\n320<\/td>\n<\/tr>\n
Other states<\/td>\n1612<\/td>\n1850<\/td>\n238<\/td>\n<\/tr>\n
Difference<\/td>\n328<\/td>\n411<\/td>\n83<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The difference-in-differences estimate, which is a more accurate measure of the policy’s impact, is 83. And if we run all this through a regression model, we can see if 83 is a significant difference between these groups.<\/p>\n

It is not:<\/p>\n

\"\"<\/a><\/p>\n

Using this more appropriate design would likely yield smaller impacts than those reported in the recent evaluation. And these small impacts likely wouldn’t be distinguishable from zero.<\/p>\n

I wanted to share this brief back of the envelope illustration for two reasons. First, I am working on a project examining Tennessee’s PBF policy and the only “impact” we are seeing is in the community college sector (more certificates). We are not finding the same results in associate’s degree or bachelor’s degree production. Second, it gives me an opportunity to explain why research design matters in policy analysis. I don’t pretend to be a methodologist or economist; I am an applied education researcher trying my best to keep up with social science standards. Hopefully this quick post illustrates why that’s important.<\/p>\n","protected":false},"excerpt":{"rendered":"

Tennessee began implementing performance-based funding in 2011 as part of its statewide effort to improve college completions. The figure below uses IPEDS completion data plotting the average number of bachelor’s degrees produced by each of the state’s 9 universities over time. One could evaluate the policy by comparing outcomes in the pre-treatment years against those […]<\/p>\n","protected":false},"author":22,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[1],"tags":[],"class_list":["post-396","post","type-post","status-publish","format-standard","hentry","category-community-college-research"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/posts\/396","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/comments?post=396"}],"version-history":[{"count":0,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/posts\/396\/revisions"}],"wp:attachment":[{"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/media?parent=396"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/categories?post=396"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/web.education.wisc.edu\/nwhillman\/index.php\/wp-json\/wp\/v2\/tags?post=396"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}