Terms of Reference for carrying out follow-up study of the New Ukrainian School reform implementation

Learning Together is a four-year collaborative project between Ukraine and Finland. The project is funded by the Finnish Ministry for Foreign Affairs and the European Commission. The Finnish consortium in charge of the project implementation is led by FCG International Ltd. The project started in August 2018 and will continue till July 2022. 

The project focuses on supporting the New Ukrainian School (NUS) reform, especially in the primary education, and is designed around three main clusters and cross-cutting elements. These clusters are (1) teacher preparation, (2) education promotion, and (3) education environment.

Learning together is now seeking to contract a qualified and experienced research organization/NGO which could carry out annual follow up study that will provide data on awareness on and perception of the NUS reform and its impact on schools. The purpose of this Terms of Reference (TOR) is to call for competitive service providers to carry out this research. This assignment focuses on carrying out research to produce reliable data about the Project’s results and the reform on which the decision makers can rely and further develop the implementation activities.  

1. 1.     Introduction

The NUS reform piloting started in 129 schools in September 2017. In those schools all the first-graders followed new education standard, curricula and teaching methods. New education environments were developed and teachers were specially trained for new teaching approaches. In 2018 the NUS reform was rolled out nationwide.

In 2019, Ministry of Education and Science (MoES) launched a study to find out results of NUS reform in everyday teaching and learning. The baseline study was conducted in autumn 2019 among 149 third grade teachers’, 897 third grade students’, 149 school principals’ and 2497 parents. It also aimed to find out the target audiences’ perceptions about the school reform. Total of 25 pilot schools and 124 “normal” schools were sampled to participate in this study.  Published results of this survey can be found by this link: https://bit.ly/3wVegyU .

For further implementation of the reform, it is planned to carry out follow-up studies among the same target audiences in 2021. It is expected that about 75% of the questionnaire items used in the study will remain the same in order to follow the trends.  Additionally, monitoring study on NUS awareness and perception will be conducted among general public and local officials.

Target groups: 3 grade teachers, 3 grade pupils, parents, school principals, general population, local educational administrations.

1. 2.     Objective

We are looking for an experienced and reliable research organization/NGO that can conduct the follow-up monitoring study in 2021. The major goal of this study is to support decision makers by providing reliable data about the level of implementation and communication of NUS. This includes also pragmatic recommendations how to further enhance the implementation. The survey will also provide monitoring data on the Project’s KPI’s, as follows:

• % of 3rd graders’ parents with positive attitudes towards NUSH reform;
• % of educational professionals with positive attitudes towards NUSH reform;
• % of citizens who are aware about the New Ukrainian School (NUS) reform;
• % of teachers and school principals and/or deputy principals who are aware of the newly approved teacher professional standards;
• % of school principals and/or deputy principals and teachers who are aware on the professional standard of a principal of general secondary education institution.

The main research question is: what is the level of implementation of NUS contents in the schools?  Thus, the study should focus on educational contents, teaching approaches and knowledge, skills and competences that students learn.

Some questions used in the first-round survey were:

• What kind of teaching methods and approaches teachers are using?
• How the educational contents have been integrated in teaching?
• Which soft / generic skills have been developed and how?
• How students collaborate, distribute tasks and share responsibilities during group work?
• How teacher-students interaction works in the classroom?
• Is there a relationship between the formation of students soft/generic skills and teachers’ methods of work, style of pedagogical interaction?
• Are students feeling comfortable and safe at the school?
• How the pedagogy of partnership is implemented in practice?
• How the teacher communicates with students and parents?
• What parents think about New Ukrainian School reform?
• To what extent are the concepts of the New Ukrainian School supported by parents?

3. Scope of Work

  The Service Provider will be contracted to:

3.1   Design, implement, analyze and report on the results of the monitoring study, including:

• Develop the research tools (including survey questionnaires and observation) based on those used in the baseline 2019 study.
• Develop the survey design (representative samples of target audiences, so that geographical, administrative-territorial, and school type related factors will be considered) in the size comparable with sample sizes of baseline 2019 survey (more details here https://bit.ly/3eHzFmD) that will allow to trace changes in 5%. Sample for online survey of 3^rd grade teachers should be enlarged to include all such teachers in the selected schools. Sample of schools should be based on data of State Scientific Institution ‘Institute of Educational Analytics. A separate analysis for the group of pilot schools in this wave is not planned, they should be presented in proportion to their number in the general population.
• Pretest the study instruments and procedures.
• Organize training for interviewers and experts of classroom teaching evaluation.
• Organize data collection.
• Analyze the data, including comparisons with baseline 2019 data, and by sex, between the types of area (urban/ruran), settlement sizes.
• Develop a technical report.
• Produce study report targeted for various audiences - MoES, local authorities, school administrations, teachers, parents, students, general public.

Planned study components and methods:

• Study of soft/generic skills of 3^rd-grade students

Methods:

1)     observation of students’ group work - 2 competence-based tasks (structured and creative) for a group of 6 randomly selected students; observation by two trained experts, filling in an evaluation form with 15 closed questions.

2)     self-administered questionnaire for students (6-10 items).

• Study of teaching methods in the classes

Method: observation of 3rd grade teachers’ performance in the class during a day; two trained experts, filling in evaluation forms – to fix teachers’ statements and coding them by 10-15 types.

• Study of 3^rd grade students’ work in the classes

Methods:

1)     observation of students’ work in the class during a day; two trained experts, filling in monitoring forms on types and formats of work used and time spent on each type/format.

2)     self-administered questionnaire for students – around 30 questions, 2/3 of them closed-end.

• Online survey of teachersUp to 30 question items – on their attitudes regarding NUS reform and teaching in primary school.
• Online survey of 3^rd-graders’ parents

Up to 40 question items, 2 of them open-ended.

• Online survey of school principals

Up to 30 question items – on their attitudes regarding NUS reform and their needs and proposals.

• Survey of general population

Up to 5 closed-end questions, their awareness and attitudes regarding NUS reform. Participation in an omnibus-type survey is envisioned.

Planed sample size – 1000.

• Survey of local authorities

Phone/online representative survey of local educational authorities, up to 15 question items, 3 of which open-ended.

Planed sample size – 600.

 4. Expected Deliverables

The Service Provider will submit the following deliverables:

4.1. Analytical framework (Description of data that will be collected in the survey, for what purpose, why it is important to collect those and how these things will be reported, where and to whom).

4.2. Development of study tools in English and Ukrainian (about 75% of the instruments and questionnaire items will be repeated from the previous baseline study) including pretesting. The questionnaires should include items for measuring Project’s Key Performance Indicators, NUS Reform Perception Key Performance Indicators as set in the Communication Strategy 2019-2022 of MoES (understanding of the reform, understanding of particular messages of the campaigns, acceptance of the reform in the target groups).

4.3. Development of study procedures.

4.2. Technical report, where study design, including sampling, weights, QM procedures, data collection procedures will be explained in detail in Ukrainian and English.

4.3. Implementation of the study in 2021.

4.4. Inputting the data, cleaning the data, in the created databank (MS Excel/SPSS).

4.5. Analyzing the data and producing results (tables and figures) accordingly to Analytical framework described in 4.1. including calculation of the Project and NUS Reform Perception Key Performance Indicators.

4.6. Draw up recommendations based on the collected data for decision makers and various target audiences.

4.7. Write summary reports on targeted audiences in close collaboration with MOES, Reform Support Team (RST) and Learning Together project in Ukrainian.

The first data collection should start in October 2021.

The first analytical report should be ready by January 10, 2022.

The supervision of work will be carried out by the Project Learning Together, Ministry of Education and Science, and Reform Support Team (RST) staff members. It is expected that MoES will support this survey by an official order to be issued not later than in June 2021.

5. Budget

The budget, details of the assignment, terms and conditions will be specified in the contract between the FCG International Ltd (Contractor) and the Service Provider. Payment milestones will be based on the acceptance of the key deliverables by the Project Management Team.

6. Submission requirements

The Proposal must include the following:

6.1 A Technical Proposal not exceeding 10 pages in length including the intended approach and the planned activities in the management of the undertaking, including an example of analytical framework, and, description about item development, operational work plan with timelines and plan how the quality of the work will be monitored. Risk analysis should be also included, as well as examples of reports on the similar projects in education.

6.2 Breakdown of costs

The budget breakdown must include two separate sections: “Fees of experts” and “Other Costs”. The fees shall be defined on rates based on working days or working months. Other Costs must be broken down to correspond the Technical Proposal and Work Plan. The Budget Breakdown must be in Euros.

6.3 The Curriculum Vitae of the team members with obligatory examples of participation in similar research projects.

6.4. Official documents proving the status of the organization and the document proving the financial capacity (annual turnover and profit/loss) of the organization.

Copies of registration documents (including a copy of the certificates of state registration of legal entities which the Participant plans to apply for the provision of services, a copy of the tax certificate; an Extract from the Statute indicating the activities).

Other relevant supporting material may be attached as an annex. 

The Proposal must be in English.

7. Evaluation Criteria

Proposals will be evaluated based on Quality (90% weight) and Price (10% weight).

The quality of the proposal will be evaluated based on the following:

Organization:

• Previous experience on similar assignments, at least one implemented nationwide study /research, preferably in the education sector.
• Proven knowledge of sampling and weighting.
• Proven knowledge of the education sector or related field.

The Service Provider’s team’s qualifications:

• A minimum of five years of professional experience in education research area, monitoring or in another related field.
• Experience of working with Ukrainian governmental organizations (in providing the social oriented surveys).
• Capability and proven experience in developing and executing all-Ukrainian monitoring, survey, research.
• Network of experts capable to conduct observation and evaluation of primary school teaching and students’ work.
• Ability and proven experience in taking initiative and working effectively under pressure.
• Excellent written, oral and interpersonal skills; excellent communication skills in Ukrainian and English.
• Familiarity with the New Ukrainian School reform.

Technical Proposal:

• The quality of proposed study set up and item development procedures.
• The quality of proposed technical operations.
• The quality of the data collection procedures.
• The quality of pretesting procedures.
• Description of the relevance of the approach and implementation methodology.
• The level of planned quality control procedures.
• The relevance of the risk analysis.
• The quality of sample material (study reports, analysis, technical reports) from the previous work.

The best candidates will be invited for an interview consisting of a 20 minutes presentation and 15 minutes for Q&A.

8. Deadline for Proposals

The timeline of the tender is as follows:

In case the tenderers want to have further clarification on the competition, written questions must be sent via email ([email protected]) by August 06, 2021.

All tenderers will be provided with answers to all clarification requests by August 11, 2021.

The final application must be submitted by August 22 (including), 2021 via email to [email protected].

Please note that all written communication must be in English. Further instructions for the tender are given only from the email mentioned above. Advice obtained from other sources may be disregarded in the tender evaluation.