Using Evidence-Based Practices in Intervention Pilots

It’s a familiar story to those involved in education or social service program delivery: a new intervention has been piloted to enhance outcomes, data have been collected to varying degrees of success, and soon the funder or sponsor will expect to see a final report.  The team is hopeful that data will show the intervention’s impact, but demonstrating impact within a short window could be difficult. Hiccups may have hindered the collection of outcomes data.  Still another measurement complication might have been if the logic model for the intervention didn’t even plan for demonstrable impacts until after the final report is due.

CHS&A has come across this scenario many times in working with clients. Most recently, we worked with a state agency that had dispersed funding to 20 grantees to engage in innovative strategies so that students with disabilities would more successfully transition from school to independent living and to a job or higher education.  Our task was to measure the impact of those pilot interventions, and our evaluation team was facing the two challenges described earlier.  First, we did not have input into the data collection and reporting plan of the individual grantees in the beginning of project implementation, and we were left hoping that the data collection protocols were sufficient to yield answers to our research questions.  Second, so many of the intended impacts of these interventions were scheduled to take place after the reporting period.  Despite our keen awareness of these complications, it was early in the process when our concerns where alleviated by a key project design criterion: the client had required that the grantees base their interventions in evidence-based practices and predictors (EBPs).

The movement to adopt EBPs in health and education has been on the rise in recent decades in education and health sectors, and for good reason.  Many well-meaning service-providers have promoted ideas for interventions based on their past experiences, intuition, or the opinions of their peers and trainers, but these don’t always prove effective.  EBPs are the result of putting these ideas to a rigorous test to determine which are reliably effective.  This means that service-providers who use EBPs aren’t enacting an intervention solely because they believe it will work, but because there is empirical evidence demonstrating results in past applications.

CHS&A was pleased to find that our task was to evaluate interventions based on EBPs.  It meant that even though we only had limited access to output and short-term outcome data, and though the medium and long-term outcomes were still too far off to measure, there was still a line in the research literature that connected the interventions in question to specific longer-term outcomes.  For example, we wouldn’t know the graduation outcomes of freshmen students who received the intervention for at least three more years, but we did know that there was evidence suggesting that the EBP led to increased graduation rate among the target population.  Essentially, EBPs allow some degree of extrapolation into the future when it comes to outcomes.  To draw this link effectively, research resources should be used to ensure that the EBP-based intervention is implemented with fidelity to the original EBP description.

If you are preparing a proposal for a pilot intervention in education or health, consider basing that intervention on EBPs, especially if the reporting window is for only a year or two.  Your proposal preparation efforts are more straight-forward since the theoretical rationales for EBPs tend to be robust.  At the end of project, you will also be able to speak more confidently about potential longer-term outcomes even if you aren’t able to demonstrate them.

Information on EBPs is widely available. In order to increase the use of EBP in practitioner communities, a multitude of research and professional organizations have provided clearinghouse websites to help search EBP databases.  See the listing below for some example resources that might be a fit for your projects.  The first listing is the one that CHS&A drew on for our evaluation in post-secondary transitions.

Education

Health/Mental Health

Posted in Data Driven-Decision making | Leave a comment

Data Driven: Menu Calorie Labeling and its (Missed) Opportunities

Last month I sat down for lunch at a midwestern diner chain and asked the server what items were popular. She rattled off a couple of dishes, among them a seafood platter. It sounded tasty if a bit heavy, but then I was shocked to see listed on the menu that this platter would run me between 2,130-2870 calories! No thank you—another equally enticing dinner platter had a full 1,200 calories less!

Politico reported that in early May the FDA had delayed the implementation of a rule requiring restaurants and other food vendors to list that caloric content of items on their menu. This delay drew attention because it came just four days before the rule would go into effect, meaning that businesses had already invested time and resources into ensuring compliance.  While there was widespread agreement among those interviewed that an unstable regulatory environment is bad for business, there are broader-ranging views on whether requiring restaurants to list caloric content is worth the trouble. I argue, based on the emerging research on food choice decision-making, that calorie labeling on menus is important for creating a decision architecture that empowers consumers and encourages competitive innovation, both of which result in a benefit for population health.

Empowers consumers

Most U.S. consumers use product attributes such as cost, taste, and convenience as primary dimensions by which they compare foods and ultimately choose what to buy and eat. One thing these attributes have in common is that people can easily and immediately understand how pricey, tasty, or convenient a food is during that dining experience. Because these attributes can be so motivating, some researchers of marketing and decision-making have called them search attributes. But there are other dimensions that are relevant to consumers’ food choice—called credence attributes—which are not immediately, experientially obvious. The nutrient profile of a food is one of these credence attributes. Unfortunately for consumers, these hidden attributes are not incorporated into our mental food-choice calculus unless they are made visible in some way. As Daniel Kahneman is fond of saying in his book Thinking Fast and Slow, “What you see is all there is.”

The impact of this tendency of human thinking is that people often make decisions without taking into account all of the relevant information. In the aforementioned Politico article, the author Helena Bottemiller Evich gives a scenario in which a convenient store pizza sells for only $1 per slice. While the search attributes of convenience, cost, and to an extent taste initially paint an appealing picture, the mental calculation is incomplete until you also realize that each slice contains between 560-620 calories. This data point may make the pizza not so appealing for some consumers, and a reasonable intake for others based on their goals and lifestyles.

Of course, I’m glossing over the complex ways in which individuals are more or less likely to seek out nutrient information based on their values and knowledge. But the basic premise holds true: consumers are most empowered to make their best choice when they are given access to credence attributes through techniques like caloric labeling. This happened for me when I used caloric labeling to avoid the seafood platter.

Encourages competitive innovation

In the diverse food marketplace, it’s no secret that some food companies are better poised to benefit from menus displaying calorie information, namely those whose brands are associated with being healthy, natural and fresh. Bottemiller Evich points out that Panera Bread has been voluntarily displaying calorie information on its menus since 2010. How much better for them if competitors are forced to do the same? Surely Panera Bread is betting that the more salient health and nutrition become to the consumer, the more likely those consumers will be to seek out Panera above its competitors.

This description may raise the hackles of free-market true believers, since requiring calorie labeling for foods might appear to push consumers towards healthier products and doom some restaurants in the process. But this regulation does not pick winners and losers. Instead, it amplifies existing pressures in the marketplace to provide a healthier product. Consumer sensibilities do evolve over time, and the most successful companies adjust to that evolution. Take McDonald’s Corporation for example. News articles from 2011, 2013, 2015, and 2016 have detailed McDonald’s efforts to keep their menu relevant and appealing to consumers by moving towards healthier items or reformulating existing items. Even as they do this, fast-casual competitors who emphasize higher-quality ingredients and fresher options pull away more market-share. Calorie labeling increases the salience of healthy eating and is likely to speed food consumers’ drift towards a health-orientation.

Benefit for population health

Calorie labeling can empower consumers to make fully-informed decisions and calculate the true value of a food purchase. As markets react to these developing consumer trends towards health, they work to make their foods more attractive by delivering what consumers want. It is a data-driven decision ecosystem on full display. Now, the delay of the labeling rule means that the public will have to continue waiting for this systemic change to build momentum. In the meantime, we’ll all have to take greater individual initiative to make informed decisions about our caloric choices.

 

Dr. Andrew Menger-Ogle is a psychologist who has a researched the influence of food labeling and marketing on decision-making.  He currently provides methodological and analytical expertise at C H Smith & Associates, LLC.

Data Driven is a series of blog posts by C H Smith & Associates to focus on the use of data for everyday decision-making.

 

Posted in Community Engagement, Data Driven-Decision making, Nutrition, Program Evaluation, Public Health | Tagged , , | Leave a comment

Credit When It’s Due: Food for the Journey

Good things are happening! Ohio just won a $500,000 grant from the Lumina Foundation for the Credit When It’s Due Initiative!

The Credit When It’s Due Initiative has 12 states and multiple funders seeking to increase the “reverse award” of associate degrees to students who transferred from a two-year institution to a four-year university and earned the relevant credit for an associate degree en route to a bachelor’s degree. I was happy to be involved in writing this grant and look forward to working with Ohio’s colleges and universities so that more students get the degrees they deserve.

For many students, especially adult learners, the road to a college degree is a complicated journey. The Credit When It’s Due Initiative can be food for this journey– awarding a credential  to students by pairing credits from the community college and university to award an associate degree. There are many real challenges for students: balancing work, family, and school; scraping up enough financial aid without creating too much of a debt burden; transferring between schools and programs; and keeping confident that you can get a degree. There are thousands of students in Ohio who are so close to a degree- they have 45 credits from a community college, transferred to a university, but have no official degree in hand. With  just one more semester at the university, these students could be eligible for associate degree.  About half of the students fitting this description are over 25 years old– adults who need to know that their efforts are not in vain, particularly if the storms of life strike and they need to stop out of college or they need a confidence boost to keep going amidst trials to obtain a bachelor’s degree and more. Awarding an associate degree to students who acquired the required credits, even though a student may still be pursuing a bachelor’s degree, is a meaningful milestone and recognizes the work put in by students.

Congratulations again to the Ohio Board of Regents and participating colleges and universities. In the next two years, I hope to hear many congratulations to thousands of graduates with college degrees in hand.

For more information on Credit When It’s Due, visit

http://www.luminafoundation.org/grants/credit_when_due.html

http://www.daytondailynews.com/news/news/state-launches-new-strategy-to-help-transfer-stude/nSg5W/

Posted in Uncategorized | Leave a comment

Face the Facts, But Keep the Faith

As Hurricane Isaac hit the Gulf Coast this week on the seventh anniversary of Hurricane Katrina and the resulting failed levees, I reflect on a powerful lesson I learned about human character and organizational development in 2005 when I was placed as a staff member on loan to help Louisiana establish what is now called the Foundation for Louisiana.   I worked for six weeks in the Louisiana capitol to set up basic building blocks for the foundation while staying in an RV with co-workers and the homes of generous Baton Rouge residents. At the same time, my sister, Jewel, a Metairie, La. resident, stayed at my home with my husband in Ohio after riding out Katrina at the New Orleans airport and getting one of the first flights out when air travel resumed. Among the many impressions and lessons, one particular mantra sticks out: Face the facts, but keep the faith.

In the classic Good to Great by Jim Collins, he refers to this as the Stockdale Paradox.  Admiral Jim Stockdale was the highest-ranking United States military officer imprisoned at the “Hanoi Hilton” prisoner of war camp for eight brutal years. He was the first three-star Navy aviator to win the Congressional Medal of Honor. Stockdale said to Collins, “You must never confuse faith that you will prevail in the end– which you can never afford to lose —with the discipline to confront the most brutal facts of your current reality, whatever they might be.”

While the challenges of starting the organization were no match to the challenges many families had for rebuilding their lives, we too had to be reminded of the Stockdale Paradox. In trying to establish a philanthropy, we were faced not only with the challenges of immediate community needs from the Katrina disaster, but also with the frustrations of complex politics, an upheaval in community networks, and oh yes, another hurricane- Hurricane Rita. We had to keep steady focus on executing the tasks within our control and believe that our work was necessary for the community to overcome the devastation of the disasters.  We could not just hope that everything would work out; we had to work it out. We had to confront the facts of nature, human errors, political miscues, and tenuous infrastructure.

Since the beginning of time, generations and generations of people, communities, and organizations have survived based on their ability to confront their reality, find practical ways of making the best of what is in their control, and maintaining hope for a better future.  In your most challenging times as an organization, you need to:

  • Face the Facts: Don’t discount the outcome data; don’t trivialize the survey results; don’t pass on the feedback like a hot potato that you never want to touch again. Analyze information with your staff and board members when appropriate. Tune your ear so that you know the truth when you hear it.  Pay attention to the external factors affecting your work and adjust. If you are truly facing the facts, you will create a plan for dealing with your organizational reality. As you outline and purse this plan, expect bumps in the road. Have a strategy for dealing with more adversity if it comes.
  • Keep the Faith: You must understand and believe in the value your organization is providing. Maintain confidence that this work will continue with successful outcomes because of the value it has in the community. Stoke the passion for the organization to not only survive but thrive in its efforts.

My thoughts and prayers are with those affected by Hurricane Isaac.  May individuals, families, and the organizations that are working to meet their needs have the strength and resources to face the facts and keep the faith. May they prevail against life’s storms.

Posted in Non-profit Organizations, Organizational Performance, Voice of the Customer | Tagged , , , , | Leave a comment

Passing the torch: Welcome to Making Good Things Happen

As the world is watching the 2012 Summer Olympics in London, I kick off the Making Good Things Happen blog of C H Smith & Associates.

I hope that this blog is useful to professionals in schools and colleges, social service organizations, community development corporations, government agencies and more. This blog will promote conversation and share lessons so that organizations can better carry out their work and serve their communities with excellence.  At times this blog will serve as a…

  • Coach: Sharing best practices and trends in the field that are demonstrating promise and success.
  • Sparring Partner: Challenging you to think and act differently about the work that we do and welcoming your challenges to the assertions of the blog so that we can all improve together.
  • Cheerleader: Keeping you encouraged as a professional. Letting you know that you are not alone in the challenges you face.
  • Teammate: Proposing ways that C H Smith & Associates can be a part of your team  to help you with a winning organizational performance.

I look forward to connecting with many of you. Thank you for following!

Calista

Posted in Uncategorized | Leave a comment