Evaluating Education Innovation: Questions to Ask Before Leaving the Shore

Innovation and transformation of our current systems in schools require leaders to rock the boat, and shun the fear of change, but before pushing off from the shore, a few questions can help us navigate a much smoother journey.

While reading a post on HASTAC, authored by Sheryl Grant regarding digital badges, her comments about the Nine Questions for Evaluating Education Innovation, a report from Nesta and New Schools Venture Fund that Tony Wan writes about in EdSurge, reminded me that not all innovative ideas are worth the energy it takes to even get the boat in the water.   How can we be sure what constitutes a best practice that leads to sustainable change?



The report is based on a recent book by Michael Fullan, Stratosphere: Integrating Technology, Pedagogy, and Change Knowledge, which emphasizes that any sustainable efforts to impact education at scale must also focus on pedagogy and system change, in addition to technology.

Authors Michael Fullan and Katelyn Donnelly offer an Index that could serve as a research framework upon which developers and school administrators can evaluate the effectiveness of new products and implementation practices. The framework assesses innovations along three components--pedagogy, technology, systems change--and breaks them down into finer detail:"

Campus and district leaders would benefit from spending time upfront asking these questions about innovative efforts, in order to plot out a successful journey into the unknown.

Pedagogy

• Is it clear what the learning outcomes are? How are they measured? (Clarity and quality of intended outcome)
• What research is it based on? Has it been applied successfully elsewhere? (Pedagogy itself)
• How detailed is it? Does it offer actionable insights? (Quality of assessment platform)

System Change

• How does the provider support technology and teachers? (Implementation support)
• What are the cost savings for the school? Are there hidden costs? (Value for money)
• Can it scale virally and laterally across teachers and schools? Does this require extensive management from the center? (Whole system change potential)

Technology

• How easy is it to use? (Quality of user experience/model design)
• Is it accessible via different devices and means? (Ease of adaptation)
• How does the technology integrate with the learning environment in the classroom? (Comprehensiveness and integration)

The authors applied this rubric to a dozen products and schools, including Khan Academy, LearnZillion, Rocketship and Carpe Diem. Their initial findings suggest pedagogy and system support "are the weakest part of the triangle," especially as "entrepreneurs find it more exciting and absorbing to design and build digital innovations than to grapple with a new pedagogy, not to mention...support for implementation."