The CFPB released a report last week detailing insights used by practitioners, researchers and funders participating in or contemplating research into the effectiveness of various financial education approaches.
According to the CFPB report, a 2011 GAO study found that little research has been conducted on the most effective strategies and methods for improving financial literacy, making it difficult for policymakers, financial institutions and service providers to determine which approaches work best.
The report detailed the use of randomized controlled trials—in which units such as schools or neighborhoods are selected at random and assigned to either a treatment or control group—used to measure program impact.
“Such a process helps to make the treatment and control groups equivalent—for example, with respect to motivation, ability, knowledge, socioeconomic and demographic characteristics, etc.—at the start of the study,” the CFPB report reads. “Then, if all goes well, any differences in outcomes… observed after the intervention can be attributed to the intervention specifically.”
The report noted, however, that not all financial literacy programs are well-suited for RCT evaluations, though experts maintain that it is the “preferred way to measure a program’s effectiveness.”
Additionally, the CFPB pointed to obstacles researchers might encounter in RCT research design and implementation, but according to research experts, data collection and analysis should “not only be considered as the capstone of the research effort but be sufficiently incorporated into the planning and research design stages and expand as study data become available.”
The report recommended that researchers first use program data to determine whether the program is ideal for evaluation and design data collection tools to complement the programs’ existing tracking instruments. It also recommended that researchers expect realistic results and consult other practitioners in the interpretation of research findings.