Keywords
external independent evaluation
educational reforms
magnitude (size) of the effect
How to Cite
Abstract
The article examines the problem relating the analysis of the effectiveness of managerial, methodical and economic interventions in the education system that are aimed at improving the academic results of students at the population level, as one of the important tasks of pedagogical science. In the study, the authors assumed that the external independent evaluation that can be considered as an objective measure of the level of academic success of graduates of general secondary education institutions makes it possible to characterize the state of the education system in Ukraine at the time of the session of the external independent evaluation.
An attempt was made to find out the impact of the overall effectiveness of the measures implemented in the education system of Ukraine on the level of educational achievements of the graduates of general secondary education institutions for the period 2009-2021, based on the analysis of official external independent evaluation reports\ that are publicly available. An educational subject was chosen for such analysis, as well as the exams that graduates passed at all sessions of the external independent evaluation during this period of time, and the above-mentioned evaluation results could be found out in the official reports of the Ukrainian Center for the Evaluation of the Quality of Education. A series of external independent evaluation sessions is considered as a series of randomized uncontrolled quasi-experiments of a national scale, which are aimed at measuring a certain parameter – the level of academic success of general secondary education institutions graduates. The result of the specified quasi-experiment is the sets of data obtained as a result of conducting a certain number of external independent evaluation sessions. Based on the obtained data, an analysis of the effectiveness of the measures implemented in the education system (educational interventions) over a long period of time as to the level of academic success of graduates of general secondary education institutions was carried out using the methods of mathematical statistics.
The article concludes that during the specified period, the success rate of the graduates of the general secondary education institutions both in individual subject tests and the average success rate according to the results of the external examinations increased. It is hypothesized that one of the main factors of such an increase was the positive behavioral reaction of society to the introduction of extracurricular activities, which is a component of many educational reforms in Ukraine.
References
Жук, Ю. О., Ляшенко, О. І., Лукіна, Т. О., Ващенко, Л. С., та Полянський, П. Б. (2011). Організаційно-методичне забезпечення моніторингових досліджень якості загальної середньої освіти : монографія. Київ : Педагогічна думка. https://lib.iitta.gov.ua/2967.
Жук, Ю. О., та Науменко, С. О. (2023). Засоби та інструменти діагностики освітніх втрат. У О. М. Топузов (Ред.), Діагностика та компенсація освітніх втрат у загальній середній освіті України : методичні рекомендації (с. 26-46). Київ : Педагогічна думка. https://doi.org/10.32405/978-966-644-736-7-2023-190. https://undip.org.ua/library/diahnostyka-ta-kompensatsiia-osvitnikh-vtrat-u-zahalniy-seredniy-osviti-ukrainy-metodychni-rekomendatsii/.
Ляшенко, О. І., Жук, Ю. О., Ващенко, Л. С., Гривко, А. В., та Науменко, С. О. (2017). Тестові технології оцінювання компетентностей учнів : посібник. Київ : Видавничий дім «Сам». https://lib.iitta.gov.ua/711199/.
Ляшенко, О. І., Лукіна, Т. О., Жук, Ю. О., Ващенко, Л. С., Гривко, А. В., та Науменко, С. О. (2018). Теоретико-методичні засади побудови моніторингових систем оцінювання якості загальної середньої освіти : монографія. Київ : ТОВ «КОНВІ ПРІНТ». https://lib.iitta.gov.ua/713252/.
Ляшенко, О. І., Лукіна, Т. О., Жук, Ю. О., Ващенко, Л. С., Гривко, А. В., Науменко, С. О., та Топузова, А. В. (2019). Запровадження моніторингових систем оцінювання якості освіти на основі тестових технологій : методичні рекомендації. Київ : Педагогічна думка. https://lib.iitta.gov.ua/719870/.
Ляшенко, О. І., Лукіна, Т. О., Жук, Ю. О., Ващенко, Л. С., Науменко, С. О., та Гривко, А. В. (2014). Тестові технології оцінювання ключових і предметних компетентностей учнів основної і старшої школи : монографія. Київ : Педагогічна думка. https://lib.iitta.gov.ua/9410/.
Про освіту. Закон України. № 2145-VIII. (2017, Вересень 5). (зі змінами 2018–2023 рр.). Верховна Рада України. Законодавство України. https://zakon.rada.gov.ua/laws/show/2145-19#Text.
Про повну загальну середню освіту. Закон України. № 463-IX. (2020, Січень 16). (зі змінами 2020-2023 рр.). Верховна Рада України. Законодавство України. https://zakon.rada.gov.ua/laws/show/463-20#Text.
Alexander, D., Heaviside, Sh., Farris E., & Westat, Inc. (1998). Status of Education Reform in Public Elementary and Secondary Schools: Teachers’ Perspectives. National Center for Education Statistics. https://nces.ed.gov/pubs99/1999045.pdf.
Berends, M., Chun, JoAn, Ikemoto, G. S., Stockly, S., & Briggs, R. J. (2002). Challenges of Conflicting School Reforms: Effects of New American Schools in a High-Poverty District. Santa Monica, CA: RAND Corporation,. https://www.rand.org/pubs/monograph_reports/MR1483.html.
Bloom, B. S. (1988). Helping all children learn in elementary school and beyond. Principal, 67(4), 12–17.
Bracey, G. W. (2000). The 10th Bracey Report on the Condition of Public Education. Phi Delta Kappan, 82(2), 133-144. https://doi.org/10.1177/003172170008200210.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, New Jersey: Erlbaum. 567 p.
Courgeau, D. (Ed.) (2003). Methodology and Epistemology of Multilevel Analysis. The Netherlands: Kluwer Academic Publishers. https://doi.org/10.1007/978-1-4020-4675-9.
Harris, D. N. (2009). Toward Policy-Relevant Benchmarks for Interpreting Effect Sizes: Combining Effects With Costs. Educational Evaluation and Policy Analysis, 31(1), 3-29. https://doi.org/10.3102/0162373708327524.
Heertum, R. V., & Torres, C. A. (2011). Educational Reform in the U.S. in the Past 30 Years: Great Expectations and the Fading American Dream. Educating the Global Citizen, 3-27. https://doi.org/10.2174/978160805268411101010003.
Klein, Jo. I., & Rice, C. (2012). U.S. Education Reform and National Security. Independent Task Force Report, 68.
Kraft, M. A. (2020). Interpreting Effect Sizes of Education Interventions. Educational Researcher, 49(4), 241-253. https://doi.org/10.3102/0013189X20912798.
Ladd, H. F., & Walsh, R. P. (2002). Implementing value-added measures of school effectiveness: getting the incentives right. Economics of Education Review, 21(1), 1–17. https://doi.org/10.1016/S0272-7757(00)00039-X.
Ladwig, J., Luke, A., Hayes, D., Mills, M. D., & Lingard, R. L. (1999). The search for productive schooling: Beyond ‘school effectiveness’ and ‘authentic. Australian Association for Research in Education Conference, Melbourne, 27 Nov – 2 Dec 1999. Coldstream, Vic.: AARE.
Let’s Go Learn. (n. d.). K-12 Education Reform: A New Paradigm. https://www.letsgolearn.com/resources/education-reform/.
Quinn, D. M., Cooc, N., McIntyre, J., & Gomez, C. J. (2016). Seasonal dynamics of academic achievement inequality by socioeconomic status and race/ethnicity: Updating and extending past research with new national data. Educational Researcher, 45(8), 443–453. https://doi.org/10.3102/0013189X16677965.
Russo, F., Mouchart, M. , Ghins, M & Wunsch, G. (2006). Statistical Modelling and Causality 1 STATISTICAL MODELLING AND CAUSALITY. https://www.researchgate.net/publication/266878390.
Schochet, P. Z., Puma, M., & Deke, J. (2014). Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. http://ies.ed.gov/ncee/pubs/20144017/pdf/20144017.pdf.
Soland, J. (2016). Is Teacher Value Added a Matter of Scale? The Practical Consequences of Treating an Ordinal Scale as Interval for Estimation of Teacher Effects. Applied Measurement in Education, 30(1), 52–70. https://doi.org/10.1080/08957347.2016.1247844.
Soland, J. & Thum, Y. M. (2019). Effect Sizes for Measuring Student and School Growth in Achievement: In Search of Practical Significance (EdWorkingPaper: 19-60). Retrieved from Annenberg Institute at Brown University. https://doi.org/10.26300/b5as-wr12.
Stuart, E. A. (2010). Matching Methods for Causal Inference: A Review and a Look Forward. Statistical Science, 25(1), 1-21. https://doi.org/10.1214/09-STS313.
Tipton, E., & Olsen, R. B. (2018). A Review of Statistical Methods for Generalizing From Evaluations of Educational Interventions. Educational Researcher, 47(8), 516-524. https://doi.org/10.3102/0013189X18781522.
von Hippel, P. T., Workman, J., & Downey, D. B. (2018). Inequality in Reading and Math Skills Forms Mainly before Kindergarten: A Replication, and Partial Correction, of “Are Schools the Great Equalizer?”. Sociology of Education, 91(4), 323–357. https://doi.org/10.1177/0038040718801760.
Weiss, M. J., Bloom, H. S., Verbitsky-Sawitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How Much Do the Effects of Education and Training Programs Vary Across Sites? Evidence from Past Multisite Randomized Trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719.
Wolf, B., & Harbatkin, E. (2023). Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type. Journal of Research on Educational Effectiveness, 16(1), 134-161. https://doi.org/10.1080/19345747.2022.2071364.
Young, V. M. (2018). Assessing the Cornerstone of U.S. Education Reform. The Journal of Educational Foundations, 31(3-4), 74-99. https://eric.ed.gov/?id=EJ1212039.
Alexander, D., Heaviside, Sh., Farris E., & Westat, Inc. (1998). Status of Education Reform in Public Elementary and Secondary Schools: Teachers’ Perspectives. National Center for Education Statistics. https://nces.ed.gov/pubs99/1999045.pdf. (in English).
Berends, M., Chun, JoAn, Ikemoto, G. S., Stockly, S., & Briggs, R. J. (2002). Challenges of Conflicting School Reforms: Effects of New American Schools in a High-Poverty District. Santa Monica, CA: RAND Corporation,. https://www.rand.org/pubs/monograph_reports/MR1483.html. (in English).
Bloom, B. S. (1988). Helping all children learn in elementary school and beyond. Principal, 67(4), 12–17. (in English).
Bracey, G. W. (2000). The 10th Bracey Report on the Condition of Public Education. Phi Delta Kappan, 82(2), 133-144. https://doi.org/10.1177/003172170008200210. (in English).
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, New Jersey: Erlbaum. 567 p. (in English).
Courgeau, D. (Ed.) (2003) Methodology and Epistemology of Multilevel Analysis. The Netherlands: Kluwer Academic Publishers. https://doi.org/10.1007/978-1-4020-4675-9. (in English).
Harris, D. N. (2009). Toward Policy-Relevant Benchmarks for Interpreting Effect Sizes: Combining Effects With Costs. Educational Evaluation and Policy Analysis, 31(1), 3-29. https://doi.org/10.3102/0162373708327524. (in English).
Heertum, R. V., & Torres, C. A. (2011). Educational Reform in the U.S. in the Past 30 Years: Great Expectations and the Fading American Dream. Educating the Global Citizen, 3-27. https://doi.org/10.2174/978160805268411101010003. (in English).
Klein, Jo. I., & Rice, C. (2012). U.S. Education Reform and National Security. Independent Task Force Report, 68. (in English).
Kraft, M. A. (2020). Interpreting Effect Sizes of Education Interventions. Educational Researcher, 49(4), 241-253. https://doi.org/10.3102/0013189X20912798. (in English).
Ladd, H. F., & Walsh, R. P. (2002). Implementing value-added measures of school effectiveness: getting the incentives right. Economics of Education Review, 21(1), 1–17. https://doi.org/10.1016/S0272-7757(00)00039-X. (in English).
Ladwig, J., Luke, A., Hayes, D., Mills, M. D., & Lingard, R. L. (1999). The search for productive schooling: Beyond ‘school effectiveness’ and ‘authentic. Australian Association for Research in Education Conference, Melbourne, 27 Nov – 2 Dec 1999. Coldstream, Vic.: AARE. (in English).
Let’s Go Learn. (n. d.). K-12 Education Reform: A New Paradigm. https://www.letsgolearn.com/resources/education-reform/. (in English).
Liashenko, O. I., Lukina, T. O., Zhuk, Yu. O., Vashchenko, L. S., Hryvko, A. V., & Naumenko, S. O. (2018). Teoretyko-metodychni zasady pobudovy monitorynhovykh system otsiniuvannia yakosti zahalnoi serednoi osvity : monohrafiia. Kyiv : TOV «KONVI PRINT». https://lib.iitta.gov.ua/713252/. (in Ukrainian).
Liashenko, O. I., Lukina, T. O., Zhuk, Yu. O., Vashchenko, L. S., Hryvko, A. V., Naumenko, S. O., & Topuzova, A. V. (2019). Zaprovadzhennia monitorynhovykh system otsiniuvannia yakosti osvity na osnovi testovykh tekhnolohii : metodychni rekomendatsii. Kyiv : Pedahohichna dumka. https://lib.iitta.gov.ua/719870/. (in Ukrainian).
Liashenko, O. I., Lukina, T. O., Zhuk, Yu. O., Vashchenko, L. S., Naumenko, S. O., & Hryvko, A. V. (2014). Testovi tekhnolohii otsiniuvannia kliuchovykh i predmetnykh kompetentnostei uchniv osnovnoi i starshoi shkoly : monohrafiia. Kyiv : Pedahohichna dumka. https://lib.iitta.gov.ua/9410/. (in Ukrainian).
Liashenko, O. I., Zhuk, Yu. O., Vashchenko, L. S., Hryvko, A. V., & Naumenko, S. O. (2017). Testovi tekhnolohii otsiniuvannia kompetentnostei uchniv : posibnyk. Kyiv : Vydavnychyi dim «Sam». https://lib.iitta.gov.ua/711199/. (in Ukrainian).
Pro osvitu. Zakon Ukrainy. № 2145-VIII. (2017, Veresen 5). (zi zminamy 2018-2023 rr.). Verkhovna Rada Ukrainy.://zakon.rada.gov.ua/laws/show/2145-19#Text. https://zakon.rada.gov.ua/laws/show/2145-19#Text. (in Ukrainian).
Pro povnu zahalnu seredniu osvitu. Zakon Ukrainy. № 463-IX. (2020, Sichen 16). (zi zminamy 2020-2023 rr.). Verkhovna Rada Ukrainy. Zakonodavstvo Ukrainy. https://zakon.rada.gov.ua/laws/show/463-20#Text. (in Ukrainian).
Quinn, D. M., Cooc, N., McIntyre, J., & Gomez, C. J. (2016). Seasonal dynamics of academic achievement inequality by socioeconomic status and race/ethnicity: Updating and extending past research with new national data. Educational Researcher, 45(8), 443–453. https://doi.org/10.3102/0013189X16677965. (in English).
Russo, F., Mouchart, M. , Ghins, M & Wunsch, G. (2006). Statistical Modelling and Causality 1 STATISTICAL MODELLING AND CAUSALITY. https://www.researchgate.net/publication/266878390. (in English).
Schochet, P. Z., Puma, M., & Deke, J. (2014). Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Analytic Technical Assistance and Development. http://ies.ed.gov/ncee/pubs/20144017/pdf/20144017.pdf. (in English).
Soland, J. & Thum, Y. M. (2019). Effect Sizes for Measuring Student and School Growth in Achievement: In Search of Practical Significance (EdWorkingPaper: 19-60). Retrieved from Annenberg Institute at Brown University. https://doi.org/10.26300/b5as-wr12. (in English).
Soland, J. (2016). Is Teacher Value Added a Matter of Scale? The Practical Consequences of Treating an Ordinal Scale as Interval for Estimation of Teacher Effects. Applied Measurement in Education, 30(1), 52–70. https://doi.org/10.1080/08957347.2016.1247844. (in English).
Stuart, E. A. (2010). Matching Methods for Causal Inference: A Review and a Look Forward. Statistical Science, 25(1), 1-21. https://doi.org/10.1214/09-STS313. (in English).
Tipton, E., & Olsen, R. B. (2018). A Review of Statistical Methods for Generalizing From Evaluations of Educational Interventions. Educational Researcher, 47(8), 516-524. https://doi.org/10.3102/0013189X18781522. (in English).
von Hippel, P. T., Workman, J., & Downey, D. B. (2018). Inequality in Reading and Math Skills Forms Mainly before Kindergarten: A Replication, and Partial Correction, of “Are Schools the Great Equalizer?”. Sociology of Education, 91(4), 323–357. https://doi.org/10.1177/0038040718801760. (in English).
Weiss, M. J., Bloom, H. S., Verbitsky-Sawitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How Much Do the Effects of Education and Training Programs Vary Across Sites? Evidence from Past Multisite Randomized Trials. Journal of Research on Educational Effectiveness, 10(4), 843–876. https://doi.org/10.1080/19345747.2017.1300719. (in English).
Wolf, B., & Harbatkin, E. (2023). Making Sense of Effect Sizes: Systematic Differences in Intervention Effect Sizes by Outcome Measure Type. Journal of Research on Educational Effectiveness, 16(1), 134-161. https://doi.org/10.1080/19345747.2022.2071364. (in English).
Young, V. M. (2018). Assessing the Cornerstone of U.S. Education Reform. The Journal of Educational Foundations, 31(3-4), 74-99. https://eric.ed.gov/?id=EJ1212039. (in English).
Zhuk, Yu. O., & Naumenko, S. O. (2023). Zasoby ta instrumenty diahnostyky osvitnikh vtrat. U O. M. Topuzov (Red.), Diahnostyka ta kompensatsiia osvitnikh vtrat u zahalnii serednii osviti Ukrainy : metodychni rekomendatsii (s. 26-46). Kyiv : Pedahohichna dumka. https://doi.org/10.32405/978-966-644-736-7-2023-190. https://undip.org.ua/library/diahnostyka-ta-kompensatsiia-osvitnikh-vtrat-u-zahalniy-seredniy-osviti-ukrainy-metodychni-rekomendatsii/. (in Ukrainian).
Zhuk, Yu. O., Liashenko, O. I., Lukina, T. O., Vashchenko, L. S., & Polianskyi, P. B. (2011). Orhanizatsiino-metodychne zabezpechennia monitorynhovykh doslidzhen yakosti zahalnoi serednoi osvity : monohrafiia. Kyiv : Pedahohichna dumka. https://lib.iitta.gov.ua/2967. (in Ukrainian).
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.