What is STiR’S impact? - STIR Education

Purpose

This document summarises STiR’s impact over the past decade
It starts by offering a brief summary of our learning, before breaking down our impact at different levels (children, teachers, officials, system) with the following questions for each:
- What are we trying to achieve?
- How do we do it?
- What evidence do we have that it’s working?
- Case studies by stakeholder

Overall Learnings

We are proud of what we have achieved over the past decade. With a small team and budget, we believe we have had a disproportionately positive impact on children, teachers and the wider education systems in which we work. That said, there has been plenty of failure, learning and iterating in that time as well. There are 2 key lessons we have learned in particular:

A focus on intrinsic motivation continues to be critical. We believe our focus on motivation is a critical part of our success, particularly at scale. Our read of the global literature on teacher professional development is that where programmes don’t ‘work’ (in terms of improving children’s learning and/or changing the behaviour and habits of teachers), they do not have a sufficient focus on how and why humans behave the way they do. Our focus on motivation not only provides this focus for our own programming, but also enables us to contribute to broader global dialogue on ‘what works’ in improving teaching.
Impact will vary at scale – responding to this is what matters most. The impact we have had on children, teachers, officials and the wider system, while positive overall, is variable by geography. This is to be expected when working across education systems with large variation in aims, values, structures and capacities. In the evaluation literature, this is known as having ‘heterogeneous effects’, a phenomenon that occurs when a successful intervention is deployed in multiple contexts. This is why we have evolved to be more responsive and flexible as have grown as an organisation, to ensure that we are as responsive as we can be to these varying contexts.

Children

What are we trying to achieve?

We want to ensure that all children love learning and are prepared to thrive in an increasingly complex world
We understand this as follows:
- Children love learning: Children have highly positive attitudes to learning and display successful learning behaviours, such as: regularly attending school, feeling safe, engaging in classroom activities, showing curiosity in class
- Children are prepared to thrive: Children develop the necessary academic and social emotional skills to thrive both within school and beyond it

We care about this because:

Children are in school but not learning – a social, economic and moral disaster exacerbated by the effects of COVID-19
Gaps between the most advantaged and disadvantaged children continue to widen
What is needed to bridge these gaps is increasingly complex, and it is clear to us that we as a global education sector need to think differently to the past on how to address this problem

How do we do it?

We believe that highly motivated teachers are key to transforming learning for children, and so our work focuses on ensuring that an entire education ecosystem is conducive to teacher motivation
We know from a wealth of evidence that teachers are the most important influence on children’s learning in school. Motivation is key to unlocking their effectiveness
We believe that intrinsically motivated teachers are particularly likely to transform children’s attitudes to learning, as well as improve academic performance
Our evidence-informed teaching strategies allow teachers to role model positive behaviour making their classrooms safer and more engaging
This is particularly powerful for the most at-risk and vulnerable students in a classroom
Because we work through the existing system, we are able to ensure these changes are high impact, low cost and scalable
We currently reach 6.9 million children globally (comprising 4.7 million in India, 2.1 million in Uganda and 93,000 in Indonesia).

What evidence do we have that it’s working?

Our impact shows improvement in children’s attitudes to learning. For example:
- In Delhi over a two-year period, we saw a 14% increase in children asking questions in class using a representative sample
- In Karnataka over a two-year period, we saw a 55% increase in children’s self-esteem in class using a representative sample
- In a survey of 8,000 teachers in India, 93% of teachers said they felt their participation in STiR’s programmes led to their children being happier in class
We have seen evidence of impact on children’s academic learning. For example:
- In Arua, Uganda, we saw pass rates in the Primary Leaving Exam improve from 73% to 95%
- In the same survey of 8,000 teachers in India, 88% felt that participating in STiR networks had led to improvement in children’s reading; 85% for improvements in numeracy
This is supported by external evidence that our approach has improved student academic learning. For example:
- A 2018 RCT conducted by an external agency on our work in Delhi demonstrated significant impact on children’s numeracy levels, with scores 6% higher compared to students in non-STiR schools (with children with low prior academic achievement performing particularly well when taught by STiR teachers)
The programme is uniquely cost effective at the child level, currently costing less than £1 per child per year in Uganda, and is expected to fall to £0.40 per year by 2025/26

Case studies

Teachers

What are we trying to achieve?

We want to unleash teachers’ inner drive to be really effective in their classrooms and transform the learning of the children they teach
We understand this as follows:
- Intrinsically motivated teachers: want to improve their practice; have the professional space to reflect upon and change their practice; have a deep sense of professional pride; make thoughtful decisions about their teaching; want to be in the classroom and maximise learning time

We care about this because:

We know that teachers are the most important school-based influence on children’s learning
However, the pressures we put on our teachers to deliver are immense. Not only do we need teachers to ‘solve’ the learning crisis, but we increasingly expect them to be the brokers of complex social phenomena such as climate change, racial justice and gender equality
This pressure is reflected in increasing teacher shortages – currently, there is a global shortage of nearly 70 million teachers, or about half the required global teaching workforce
Clearly this cannot continue. Efforts to ensure that teachers are highly technically proficient must concurrently build motivation in the teaching workforce, and we believe that there is currently not sufficient attention on the latter

How do we do it?

For teachers to be intrinsically motivated, they need to feel their decisions matter, they see themselves improving, and they feel connected to their peers and the children they teach.
We believe these needs can be met through the provision of high quality professional development, composed of the following elements:
- Monthly network meetings – to help teachers reflect on how to make decisions in the classroom, improve their teaching practice, and develop a sense of deep connection to their colleagues
- Peer observation and feedback – to empower teachers to take control of their own development, to improve their teaching in context, and to both support and be supported by their colleagues
- Reflection – to enable teachers to critically reflect on their practice and take actions accordingly
Our role at the teacher level is to co-create the teacher level content with our partner governments. We ensure that content is faithful to the principles above, with specific content tailored to the needs and priorities of each geography
We currently support 245,400 teachers globally (comprising 180,000 in India, 60,000 in Uganda and 5,400 in Indonesia)

What evidence do we have that it’s working?

In Delhi, we saw a 36% increase in engagement in teacher network meetings over a 4-year period
In Karnataka, we saw a 17% increase in teachers displaying behaviours indicating curiosity and critical thinking in network meetings over a 4-year period
In Uganda, 53% of teachers felt they had created more positive classrooms as a result of their participation in STiR networks over a 1-year period
Our 2018 RCT showed STiR teachers in Delhi were significantly more likely to develop a Growth Mindset compared to non-STiR teachers (0.18 SD)
This RCT also showed STiR teachers spent 8% more time teaching than non-STiR teachers in Uttar Pradesh (equivalent to saving the government $7 in costs for every $1 invested in the programme)

Case studies

Officials

What are we trying to achieve?

We empower school leaders and district-level education officials to create the conditions in which teacher intrinsic motivation can flourish, through role-modelling intrinsic motivation
We understand this as follows:
- Intrinsically motivated officials and school leaders: are interested in developing cultures of improvement; create environments where mistakes are normalised; have a deep sense of professional pride; make thoughtful decisions about their leadership; want to spend time in schools understanding and responding to the needs of teachers

We care about this because:

Teachers do not exist in a vacuum. The professional conditions in which they work are shaped by a variety of factors – most crucially by school leaders and education officials
All research supports that investment in teachers is crucial to improved student outcomes but STiR’s decade of experience in this field shows true change must come from strengthening the whole education system
Because we work through the system, it is school leaders and officials that deliver the programme for teachers; therefore, the attitudes and behaviours of officials are core to the programme’s success

How do we do it?

To be intrinsically motivated, school leaders and officials need to understand how their actions can influence how teachers feel about work, see that these actions are making a positive impact on this over time, and feel connected to their peers and the teachers they support
We believe that these needs can be met through the same mechanisms of peer connection, observation/feedback and reflection
These mechanisms look slightly differently for both school leaders and education officials compared to teachers. Instead they have:
- Quarterly training institutes – to help school leaders and officials reflect on how to make decisions that will empower teachers; improve their instructional leadership, and develop a sense of deep connection to their colleagues
- Observation and feedback – to help school leaders and officials understand their strengths and areas for development with regards to creating the conditions for teachers to thrive (eg observation of facilitation of network meetings)
- Reflection – to support school leaders and officials to reflect on their leadership based on data and take actions accordingly
Our role at the school leader and official level is two-fold. Firstly, we co-create the training content with our partner governments, ensuring the content is faithful to the principles above. Secondly, we work directly with district and senior officials, providing coaching and wider strategic support in ensuring a culture of intrinsic motivation can flourish

What evidence do we have that it’s working?

In Delhi, over a 4-year period we saw an increase of 19% in officials’ self-esteem
In Karnataka, over a 4-year period we saw an increase of 30% of school leaders and officials demonstrating curiosity and critical thinking in how they can better support teachers
In Uganda, 100% of school leaders surveyed said they believed the coaching support they received helped them improve their effectiveness

Case studies

System

What are we trying to achieve?

Our programme ignites education systems. We work at district and national level to empower and motivate teachers, school leaders and education officials. This strengthens the whole education system and provides better outcomes for children
We understand this as follows:
- National and state officials: are highly committed to quality teacher professional development and excited by the transformational power of intrinsic motivation in its delivery, and work to ensure all aspects of the system are designed for its achievement

We care about this because:

If we are to support teachers, school leaders and officials to deliver at scale, support at the national and state level is critical
We will only ever expect to see pockets of success in a system where there is no co-ordination in policy and resource support at the national and state level

How do we do it?

We primarily through government systems, investing in relationships with key stakeholders at state and national level, in order to understand their priorities and how our approach can help achieve them – as well as build trust
We identify where responsibility for our approach is likely to be most successfully embedded within the system, and work with officials to make this a reality
We share data at the national level, to ensure decision-makers are clear on what is and is not working
We sign detailed MoUs with governments to ensure that respective roles are clearly agreed and understood
We identify opportunities to embed a commitment to intrinsic motivation in national policies
We agree a sustainability strategy from the outset, ensuring government gradually take full ownership of the approach
In order to ensure we remain nimble and responsive to the highest leverage entry points in a system, we have also begun to explore working through and with partner organisations in a technical advisory capacity where the partners’ aims are also to embed intrinsic motivation across the system. This is the model we are pursuing in pilot projects in Brazil and Ghana

What evidence do we have that it’s working?

As a result of our influence, a section on motivation was included in the National Teacher Policy in Uganda
A review commissioned by the Mastercard Foundation in 2019 found that our work in Uganda was directly supporting the government’s priorities for teaching and learning, and that officials showed commitment to the concept of teacher motivation
In Delhi, we co-created the Lifelong Learning Unit, a government cell comprised of state officials, district officials and support teachers to oversee the design and delivery of the programme
We have expanded our work exponentially at the request of our government partners. For example, in Delhi we started with a pilot of 25 teachers and now work across all 1,029 secondary schools in the city; in Uganda, we initially worked with 684 teachers and today impact more than 80,000
The technical advisory opportunities in Brazil and Ghana have arisen after the respective partner organisations in each country sought us out, based on our reputation as experts in building intrinsic motivation across systems.

Going Forward

The above gives us confidence that our work is having significant positive impact at a variety of levels and across geographies. We recognise, however, that there are gaps in terms of up-to-date, rigorous causal evidence on learning and behaviour change. With this in mind, we have re-designed our impact evaluation strategy with a plan to commission a large-scale, global quasi-experimental study which we believe will help to address this gap. We plan to begin the study in late 2023.