Project Title | : Strengthening Literacy and Numeracy in Sumba Barat Daya through Inquiry-based Learning |
Funder | : William & Lily Foundation |
Project Period | : December 2017 – December 2021 |
Evaluation Period | : September 2021 – November 2021 |
Introduction & Rationale for Evaluation
The William & Lily Foundation (WLF) is an Indonesian-based philanthropic foundation established in 2009 as an outgrowth of the philanthropic legacy of William and Lily Soeryadjaya. As an active grant-making foundation, WLF works closely with implementation partners on four focus areas: education, health, economic empowerment and enabling environment. WLF is committed to impactful and responsible interventions in its efforts to empower vulnerable and marginalized communities across Indonesia. Although the scope of WLF philanthropic endeavours is countrywide, its current geographical focus centres on eastern Indonesia, particularly on the island of Sumba.
In December 2017, WLF formed a partnership with Kuark International, in implementing a two-year program entitled Strengthening the Concept of Basic Literacy Education Curriculum through Local Potential For elementary schools in Sumba Barat Daya. This program is called Kelas Lentera. The project aims to (1) improve the quality of teacher’s teaching and students' basic literacy and numeracy skills by strengthening teachers' capacity in implementing various learning strategies and (2) improve students' cognitive development by applying a holistic approach which also focuses on motor development. The beneficiaries of this project are the school principals, teachers and students in three elementary schools in Sumba Barat Daya.
Kelas Lentera Program has evolved during the program implementation. In phase 1, Kelas Lentera focuses more on strengthening teacher capacity on inquiry-based approach and local-based utilization. In phase 2, Kelas Lentera Program not only strengthens teacher capacity on literacy and numeracy but also strengthens student capacity through brain training. In phase 3, Kelas Lentera program strengthens the quality of literacy and numeracy learning through increasing teacher capacity and also encourages sharing and learning among teachers in literacy and numeracy learning through the Teacher Working Group (Kelompok Kerja Guru, or KKG) platform.
In November 2020, WLF conducted an evaluation covering Kelas Lentera Program for Phase 1 and 2. The evaluation was aimed to confirm efficacy of the program approach, particularly with the addition of specific interventions on the second half of the program such as Brain fit activity and the use of alternative communication media (ie, WhatsApp, etc.) and to validate results and potential replication of this project. Based on the evaluation finding, this program has succeeded in improving the capacity of teachers through inquiry-based approach. The implementation of inquiry-based approach has also improved the involvement of students in the learning process through local content curriculum, utilization of local learning material, and strengthening basic literacy foundations such as brain training. In addition, the teaching and learning activities have also succeeded in improving student’s soft skills.
To ensure the impact of the program on the beneficiaries, the Kelas Lentera project was then continued in two previous elementary schools. In this third year period, the Kelas Lentera also added new program objectives, namely improving awareness of teachers to share their best practices in teaching literacy and numeracy in the KKG (Kelompok Kerja Guru) forum. As per WLF’s policy, this project is subject to external independent evaluation.
The Kelas Lentera project's interventions include providing training on literacy and numeracy learning methods to the teachers in target schools, providing teacher coaching for the development and implementation of literacy and numeracy pedagogy products. This project also utilizes the KKG platform to facilitate the process of sharing best practices in literacy and numeracy learning with other teachers in non-assisted schools. As this project is nearing completion, WLF plans to conduct a comprehensive final evaluation that will provide a 360 degree outlook not only on project performance measured against its intended outcomes, but also on the effectiveness of project management and implementation process, the appropriateness of chosen interventions, as well as to evaluate whether this project has applied the value for money principal. As per WLF’s policy, this project is subject to external independent evaluation. WLF is therefore seeking individual or a team of evaluators to conduct this evaluation.
Purpose, Scope and Users of Evaluation
The first objective of this external and independent evaluation is to provide a comprehensive evaluation on the project management and implementation process performed by the implementing partner, in this case Kuark International; particularly on how the project was able to meet the agreed upon project objectives, outcomes and outputs, as well as to evaluate whether the project has been carried out in a cost-efficient manner. The second objective is to analyse and document the challenges faced, lessons learned and, if any, best practices produced. Last but not least, this evaluation also aims to understand the efficiency and effectiveness of the grants and technical support WLF provided to the implementing partner during the lifetime of the project.
The evaluation will cover the project's various components (such as the expected outcomes, outputs, and activities) as reflected in the project document as well as subsequent modification and alterations made during its implementation. The evaluation will cover the Kelas Lentera program from phase 1 until phase 3. As a key objective of this project has been to improve the quality of literacy and numeracy through inquiry based learning approach, this evaluation is also expected to cover sustainability of the learning approach as well as extraction of inquiry based approach learning in the target schools. The main user of findings and recommendations from this evaluation will be the William and Lily Foundation. Other potential users may include but not limited to the Foundation’s implementing partners and relevant stakeholders as necessary.
Methodology
The evaluation will be based on a participatory approach, involving a wide range of selected key stakeholders and beneficiaries. The evaluation will use both primary and secondary data sources for probing the progress and for generating evidence in support of the findings. Overall, the evaluation approach should refer to OECD-DAC criteria (considering factors such as relevance, coherence, effectiveness, efficiency, sustainability and impact). To the extent possible, quantitative and qualitative data will be collected, validated, triangulated and analysed. The evaluation process will include desk reviews, interviews and observation. The proposed methodology is not fixed, however, and can be adjusted by the Evaluator if considered necessary, in accordance with the scope and purpose of the evaluation and in consultation with the Foundation team.
Evaluation Criteria and Suggested Questions
The Project will be evaluated against criteria such as its relevance to organization mission and objectives of organization, the validity of project design, project effectiveness, the efficiency of resource use, the impact orientation and sustainability. To summarize, the external independent evaluation should address the following key questions:
1. Relevance and value to organization strategy and mission statement
● To what extent does the project design take into account the needs of beneficiaries and stakeholders?
● To what extent does the project design align with WLF’s mission and objectives?
2. Validity and relevance of design
● Are the planned project objectives, outcomes, outputs and activities aligned?
3. Project achievements and effectiveness
● To what extent has the project achieved their stated objectives?
● What are the main factors – internal to the project and external – that have hindered the project’s ability to reach the desired results thus far?
4. Efficiency of resource use
● To what extent have technical and administrative support been sufficient and adequate to meet project objectives?
5. Effectiveness of grant management; and
● To what extent have the WLF grant management support been adequate to meet project objectives?
6. Impact orientation and sustainability in relevance to the target community
● To what extent are planned results of the project likely to be sustained and/or scaled up and replicated by stakeholders?
● What further concrete steps could be taken to increase the sustainability of the results?
Evaluator’s Responsibilities and Deliverables
1. Key Responsibilities
● The design, planning and implementation of the evaluation and the write-up of the evaluation report, using an approach agreed by WLF and for delivering in accordance with the WLF’s specifications and timeline
● Consulting and liaising, as required, with WLF, stakeholders and partners to ensure satisfactory delivery of all deliverables
● Making herself/himself available, if required, to take part in briefings and discussions, online or other venue, on dates to be agreed
2. Key Deliverables
● Inception Report with Methodology
The inception report should detail the evaluator’s understanding of what is being evaluated and why, showing how each evaluation question will be answered by way of: proposed methods; proposed sources of data; and data collection procedures. The inception report should also include an evaluation framework matrix, a stakeholder list, proposed interview tools/guides, proposed schedule of tasks, activities, deliverables, and outline of the final report.
● Draft of Evaluation Report
The Evaluator will submit a draft of the evaluation report to WLF. The recommendations, lessons learned, and emerging good practices should be documented in the draft evaluation report.
● Presentation of Draft Report
A presentation should be prepared for the WLF on the draft report, to be used during the debriefing.
● Final Evaluation Report
To be submitted to WLF as per report outline and timeline. The quality of the report will be determined based on quality standards defined by WLF. The report should also, as appropriate, include specific and detailed recommendations by the Evaluator based on the analysis of information obtained. The report should also include a specific section on lessons learned and good practices that are relevant for better performance and delivery for future projects or could be replicated or those that should be avoided.
Team Composition
The evaluation will be conducted by a team comprised by a team leader and a team member
Proposed Time Line
The evaluation is foreseen to be undertaken in the time period, September 2020 to November 2021 with the aim to submit the final evaluation report to WLF no later than mid December 2021. The actual implementation of this evaluation will depend on the pandemic situation where movements are restricted.
Proposal Submission Criteria
The proposed evaluation design must take into account Covid-19 health protocol and alternative methods of data collection during the period of restricted movement. The proposal should at least contain the following information:
1. Proposed methodology and approaches for the evaluation design
2. Proposed team composition (clearly stating roles and responsibilities of each team member)
3. Financial proposal to conduct the evaluation, complete with breakdown of man-hours or days required to conduct each stage of the preparation, data collection tools development, data collection and analysis, field visit/travel, report writing, and presentation
4. Most current Curriculum Vitae, highlighting related experiences and skills to the required scope of work
5. Sample of previous evaluation reports
Essential Requirements:
1. A bachelor’s degree in education sciences or any relevant social sciences, Master’s degree is preferred
2. A minimum of 5 years of experience conducting monitoring and evaluation of development programs in impoverished regions in Indonesia, particularly in carrying basic evaluation methodology
3. Demonstrable research and strong analytical skills
4. Demonstrable strong knowledge on basic literacy and numeracy education in elementary schools, having strong pedagogic background on literacy and numeracy particularly for primary school and knowledge of inquiry-based methods
5. Experience in conducting impact measurement on literacy and numeracy in early grade students.
6. Strong interpersonal skills in a multicultural setting, experienced in engaging principals, teachers, students in elementary schools and key government officials at subnational levels
How to Apply
Interested organizations/consultants are invited to send complete application (including cover letter, proposal, budget, most updated CVs of team members, and samples of previous evaluation reports) through the following link, https://tinyurl.com/Evaluasi-KL before 20 August 2021 at 23:58 Jakarta time at the latest.