What did we intend to learn?
The world is undergoing rapid social, economic and technological changes. Children today need to develop a love of learning that will last their whole life. At STiR Education, we support education systems to reignite intrinsic motivation in every teacher and official, so that they can role-model the foundations of lifelong learning for every child.
Our learning from an ongoing longitudinal study in India and Uganda suggests that officials and teachers are changing their behaviour at a large scale. Officials are actively observing teachers and providing them with feedback, and teachers are consequently trying out new evidence-based practices. The big challenge ahead is to deepen the quality of these behaviour changes. Our hypothesis is that by supporting officials and teachers to reflect more deeply on what’s going well and what teachers could do better, we can stimulate more nuanced coaching conversations and drive consequent changes in teaching practices.
We believe that technology can play an exciting and critical role in driving this behaviour change at a very large scale. Having explored various technology platforms, we decided to test our hypothesis using a WhatsApp chatbot. We chose WhatsApp because it is already widely used by the teachers and officials we work with, and they’re therefore more likely to use this compared to other mobile apps.
How does the chatbot work?
The WhatsApp chatbot runs on an encrypted WhatsApp platform. The chat conversation is not shared with any other person as it is a 1:1 conversation with the bot. It is available 24/7, so users can start their chats anytime whenever they are available.
We chose to pilot the chatbot in Delhi and Uganda to keep it manageable and cost effective, as we didn’t know how the technology would respond to our design. In each location, we piloted it with 30 officials who were connected through WhatsApp.
In the following, you will find the key steps of our design in the chatbot:
1. Opt-in and registration:
It was very simple and easy to join the chatbot unlike other mobile apps. We shared the chatbot joining link with the officials using other WhatsApp groups where all the officials were already connected. As you can see in the animation, after joining the chatbot they receive an introduction message and then follow the instructions and codes to fill in their basic details about their role in the programme. Based on their role and location, the chatbot sends them the relevant details and instructions to follow.
2. Reflection sharing (network meeting):
After the registration, the chatbot takes them to the main menu page where they can choose the activity and add their reflections on recent observations. The first step is to fill in their reflections about the network meeting they observed. They just type a code and then the chatbot starts responding with the reflective questions one by one. They can respond to each question in Y and N. At the end of the survey, the chatbot gives them a summary of their responses to confirm that they have marked all the questions correctly.
3. Video nudges
The chatbot then analyses their reflection data and automatically sends a video nudge of a model example of behaviour with a follow-up probing question. The officials then use this video and learn about the underlying principle of the particular practice which they did not observe in the network meeting. The official and their colleague then reflect on what went well and areas of improvement during their next feedback meeting, and discuss how best they can use this video content to improve that behaviour.
So every month officials observe the network meeting, fill in their reflections and coach their colleagues using the video message. Month by month, we observe their behaviours and see how videos and probing questions help them in improving their practices. In the first pilot, the chatbot tool worked well, but due to a reduced number of activities we could not get enough data to understand the impact of nudges. We also had challenges with officials’ access to the internet in Uganda. In the following section, we have summarised our key learnings from the phase 1 of the pilot.
What have we learned?
1) WhatsApp is indeed embedded in the existing behaviour of officials, and the chatbot was therefore a user-friendly tool that doesn’t force them to download another app in a crowded technology landscape. Some of the key features of the chatbot include:
- Easy to opt-in and get engaged
- Instant reply
- Accessible 24/7
- Simple flow and easy codes to follow (with minimum support)
- Low internet usage
2) The evidence-based nudges that we shared with officials helped them to have higher quality coaching discussions, which was the key aspect we aimed to test in this pilot.
- Short and instant video nudges
- Simple one-pager of monthly insights based on their reflections
3) Looking forward, we need to better understand whether these higher quality coaching discussions do indeed lead to consequent behaviour change. Key areas to be improved:
- Collecting more behavioural data by adding more measures and geographies
- Adding more videos on good and bad practices
- Creating content in local language (Hindi, Tamil, Kannada)
Next phase of the pilot
In phase 2, we are planning to improve and test the following areas:
- Overall, we would like to learn more about the changes in behaviours by adding more samples, activities and videos. E.g. once people watch the video, what do we hope happens? We hope to see engaging and focused conversations among system officials that in turn influence what teachers do in the classroom.
- We learned that video messages are more impactful than text messages. We will be creating a bank of short videos on the best and not so good practices for a range of topics.
- We will be developing all the content in local languages so that officials can choose their preferred language.
- We are planning to create a dashboard or a report section where officials can access their data and insights of the previous months, so they can become more action-oriented and focus on planning during their developmental conversations.
We will be piloting the chatbot in Delhi, Karnataka and Tamil Nadu in India from August 2021 with thousands of officials. Due to the lack of internet connectivity in Uganda, we’re exploring other mediums of technology for future pilots.
To learn more about this project or if you have any suggestions for further improvements, please contact Perwinder Singh, Lead Senior Manager for Monitoring and Evaluation, at psingh(at)STiReducation.org.
