Speaking the Truth with Conflicting Data

The C H Smith & Associates team joined with our former client representative Dr. Susan Zelman of the Ohio Department of Education to present at the 2018 annual American Evaluation Association conference in Cleveland, Ohio on November 3.  In a presentation entitled “Speaking Truth with Conflicting Data”, we shared with other professionals in the evaluation community about our experiences of evaluating the first two cohorts of the Ohio Straight A Fund (RIP).  So while the Straight A Fund is no longer alive, we did not want its lessons to be left in the graveyard. Our aim was to start a discussion in which all of us could acknowledge some challenging aspects of the field and get unstuck in our search for solutions.

The truth is that even when data are collected effectively and appropriately, the story they tell are not always clear.  We illustrated this through sharing our encounters with messy, complex, and conflicting data in the world of K-12 education evaluation.  For this problem, we relied on a storytelling solution.  We made a decision that it was appropriate to let the conflicting data co-exist to give a complex answer on project efficacy that arose from data triangulation.  In practice, this approach meant detailing the school district’s data dilemma in an insert to the full technical report, with a goal to build empathy in the client for the sub-grantee.

Besides providing the typical “lessons learned” from the perspective of the evaluator, one aspect of our discussion that was notable was the participation of our client, who commented on her reaction to the challenges that came up and the solutions that we chose.  It seemed that the audience of evaluators found this perspective illuminating, with a few attendees making comments about how these views related to their own work.

It is so often such a positive experience to look back on past work with colleagues and to share knowledge within a professional community.  Thanks to Dr. Susan Zelman for partnering with us and to all the other generous presenters at AEA2018!



About Andrew Menger-Ogle, PhD

Dr. Andrew Menger-Ogle is Research and Evaluation Associate at C H Smith & Associates, LLC. Andrew currently provides methodological and analytical expertise for program evaluations.
This entry was posted in Data Driven-Decision making, Program Evaluation and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *