Are open educational resources (OER) and practices (OEP) efective in improving learning achievement? A meta-analysis and research synthesis

Bibliographic References

• Allen, G., Guzman-Alvarez, A., Smith, A., Gamage, A., Molinaro, M., & Larsen, D. S. (2015). Evaluating the efectiveness of the open-access ChemWiki resource as a replacement for tr*aditional general chemistry textbooks. Chemistry Education Research and Practice, 16(4), 939–948. https://doi.org/10.1039/c5rp00084j
• Bali, M., Cronin, C., & Jhangiani, R. S. (2020). Framing open educational practices from a social justice perspective. Journal of Interactive Media in Education, 2020(1), 1. https://doi.org/10.5334/jime.565
• *Basu Mallick D., Grimaldi P. J., Whittle J., Waters A. E., & Baraniuk R. G. (2018). Impact of OER textbook adoption on student academic outcomes. Paper presented at the 15th Annual Open Education Conference, Niagara Falls, NY.
• Bedenlier, S., Bond, M., Buntins, K., Zawacki-Richter, O., & Kerres, M. (2020). Facilitating student engagement through educational technology in higher education: A systematic review in the feld of arts and humanities. Australasian Journal of Educational Technology, 36, 126–150. https://doi.org/10.14742/ajet.5477
• Borenstein, M. (2022). Comprehensive meta-analysis software. In M. Egger, J. P. T. Higgins, & G. D. Smith (Eds.), Systematic reviews in health research: Meta-analysis in context (pp. 535–548). Wiley. https://doi.org/10.1002/9781119099369.ch27
• Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2010). A basic introduction to fxed-efect and randomefects models for meta-analysis. Research Synthesis Methods, 1(2), 97–111. https://doi.org/10.1002/jrsm.12
• Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to Meta-Analysis (2nd ed.). John Wiley & Sons. https://doi.org/10.1016/b978-0-12-209005-9.50005-9
• Bozkurt, A., Gjelsvik, T., Adam, T., Asino, T. I., Atenas, J., Bali, M., ... Weller, M., Xiao, J., & Zawacki-Richter, O. (2023). Openness in Education as a Praxis: From Individual Testimonials to Collective Voices. Open Praxis, 15(2), 76–112. https://doi.org/10.55982/openpraxis.15.2.574
• Burçin, Ö. N. E. R. (2022). Evaluation of statistical power in random efect meta analyses for correlation efect size. Sakarya University Journal of Science, 26(3), 554–567. https://doi.org/10.16984/saufenbilder.1089793
• Butcher, N. (2015). Basic guide to open educational resources (OER). Commonwealth of Learning (COL). https://doi.org/10.56059/11599/36
• Cafri, G., Kromrey, J. D., & Brannick, M. T. (2009). A sas macro for statistical power calculations in metaanalysis. Behavior Research Methods, 41(1), 35–46. https://doi.org/10.3758/brm.41.1.35
• Cafri, G., Kromrey, J. D., & Brannick, M. T. (2010). A meta-meta-analysis: Empirical review of statistical power, type I error rates, efect sizes, and model selection of meta-analyses published in psychology. Multivariate Behavioral Research, 45(2), 239–270. https://doi.org/10.1080/00273171003680187
• Chaudhary, P., & Singh, R. K. (2022). A meta analysis of factors afecting teaching and student learning in higher education. Front. Educ., 6, 824504. https://doi.org/10.3389/feduc.2021.824504
• Chen, Z., Chen, W., Jia, J., & An, H. (2020). The efects of using mobile devices on language learning: A meta-analysis. Educational Technology Research and Development, 68(4), 1769–1789. https://doi.org/10.1007/s11423-020-09801-5
• Cheung, A., & Slavin, R. E. (2016). How methodological features of research studies afect efect sizes. Educational Researcher, 45(5), 283–292. https://doi.org/10.3102/0013189X16656615
• *Chiorescu, M. (2017). Exploring open educational resources for college algebra. International Review of Research in Open and Distributed Learning, 18(4), 50–59. https://doi.org/10.19173/irrodl.v18i4.3003
• *Choi, Y. M., & Carpenter, C. (2017). Evaluating the impact of open educational resources: A case study. Portal: Libraries and the Academy, 17(4), 685–693. https://doi.org/10.1353/pla.2017.0041
• Clark, R. E. (1994). Media will never infuence learning. Educational Technology Research and Development, 42(2), 21–30. https://doi.org/10.1007/BF02299088
• *Clinton, V. (2018). Savings without sacrifces: A case study of open textbook adoption. Open Learning: THe Journal of Open, Distance, and e-Learning, 33(3), 177–189. https://doi.org/10.1080/02680513.2018.1486184
• Clinton, V., & Khan, S. (2019). Efcacy of open textbook adoption on learning performance and course withdrawal rates: A meta-analysis. AERA Open, 5(3), 2332858419872212. https://doi.org/10.1177/2332858419872212
• Cohen, J. (1960). A coefcient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
• *Colvard, N. B., Watson, C. E., & Park, H. (2018). The impact of open educational resources on various student success metrics. International Journal of Teaching and Learning in Higher Education, 30(2), 262–276.
• Denden, M., Tlili, A., Chen, N. S., Abed, M., Jemni, M., & Essalmi, F. (2022). The role of learners’ characteristics in educational gamifcation systems: A systematic meta-review of the literature. Interactive Learning Environments. https://doi.org/10.1080/10494820.2022.2098777
• Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017). Low statistical power in biomedical science: A review of three human research domains. Royal Society Open Science, 4(2), 160254. https://doi.org/10.1098/rsos160254
• Ehlers, U.-D. (2011). Extending the territory: From open educational resources to open educational practices. Journal of Open Flexible and Distance Learning, 15(2), 1–10. http://www.jofdl.nz/index.php/JOFDL/index
• *Engler, J. N., & Shedlosky-Shoemaker, R. (2019). Facilitating student success: The role of open educational resources in introductory psychology courses. Psychology Learning & Teaching, 18(1), 36–47. https://doi.org/10.1177/1475725718810241
• Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*power 3: A fexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146
• *Feldstein DPS, A. P., Martin, M., Hudson, A., Warren, K., Hilton III, J., & Wiley, D. (2012). Open textbooks and increasedstudent access and outcomes. European Journal of Open, Distance and E-Learning. https://old.eurodl.org/?p=archives&year=2012&halfyear=2&article=533
• Fischer, L., Hilton, J., III., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of open textbook adoption on the learning outcomes of post-secondary students. Journal of Computing in Higher Education, 27(3), 159–172. https://doi.org/10.1007/s12528-015-9101-x
• Fortney, A. (2021). OER textbooks versus commercial textbooks: Quality of student learning in psychological statistics. Locus: The Seton Hall Journal of Undergraduate Research, 4(1), 4.
• Garzón, J., Pavón, J., & Baldiris, S. (2019). Systematic review and meta-analysis of augmented reality in educational settings. Virtual Reality, 23(4), 447–459. https://doi.org/10.1007/s10055-019-00379-9
• Geissbühler, M., Hincapié, C. A., Aghlmandi, S., Zwahlen, M., Jüni, P., & da Costa, B. R. (2021). Most published meta-regression analyses based on aggregate data sufer from methodological pitfalls: A meta-epidemiological study. BMC Medical Research Methodology, 21, 123. https://doi.org/10.1186/s12874-021-01310-0
• *Grewe, K., & Davis, W. P. (2017). The impact of enrollment in an OER course on student learning outcomes. The International Review of Research in Open and Distributed Learning. https://doi.org/10.19173/irrodl.v18i4.2986
• Grifn, J. W. (2021). Calculating statistical power for meta-analysis using metapower. The Quantitative Methods for Psychology., 17(1), 24–39. https://doi.org/10.20982/tqmp.17.1.p024
• Grifths, D., Burgos, D., & Aceto, S. (2022), Credentialing learning in the European OER Ecosystem. Retrieved July, the 14th, 2023, from https://encoreproject.eu/2022/09/06/credentialing-learning-in-the-european-oerecosystem/
• Griggs, R. A., & Jackson, S. L. (2017). Studying open versus traditional textbook efects on students’ course performance: Confounds abound. Teaching of Psychology, 44(4), 306–312. https://doi.org/10.1177/0098628317727641
• Grimaldi, P. J., Basu Mallick, D., Waters, A. E., & Baraniuk, R. G. (2019). Do open educational resources improve student learning? implications of the access hypothesis. PLoS ONE. https://doi.org/10.1371/journal.pone.0212508
• *Grissett, J. O., & Hufman, C. (2019). An open versus traditional psychology textbook: Student performance, perceptions, and use. Psychology Learning & Teaching, 18(1), 21–35. https://doi.org/10.1177/1475725718810181
• *Gurung, R. A. (2017). Predicting learning: Comparing an open educational resource and standard textbooks. Scholarship of Teaching and Learning in Psychology, 3(3), 233–248. https://doi.org/10.1037/stl0000092
• *Hardin, E. E., Eschman, B., Spengler, E. S., Grizzell, J. A., Moody, A. T., Ross-Sheehy, S., & Fry, K. M. (2019). What happens when trained graduate student instructors switch to an open textbook? A controlled study of the impact on student learning outcomes. Psychology Learning & Teaching, 18(1), 48–64. https://doi.org/10.1177/1475725718810909
• *Harvey, P., & Bond, J. (2022). The efects and implications of using open educational resources in secondary schools. The International Review of Research in Open and Distributed Learning, 23(2), 107–119. https://doi.org/10.19173/irrodl.v22i3.5293
• Hedges, L. (1981). Distribution theory for glass’s estimator of efect size and related estimators. Journal of Educational Statistics, 6(2), 107–128. https://doi.org/10.3102/10769986006002107
• Hedges, L., & Olkin, I. (1985). Statistical methods for meta-analysis. Academic Press.
• *Hendricks, C., Reinsberg, S. A., & Rieger, G. W. (2017). The adoption of an open textbook in a large physics course: An analysis of cost, outcomes, use, and perceptions. International Review of Research in Open and Distributed Learning,18(4), 78–99. https://doi.org/10.19173/irrodl.v18i4.3006
• Hilton, J. III. (2016). Open educational resources and college textbook choices: A review of research on efcacy and perceptions. Educational Technology Research and Development, 64(4), 573–590. https://doi.org/10.1007/s11423-016-9434-9
• Hilton, J., III. (2020). Open educational resources, student efcacy, and user perceptions: A synthesis of research published between 2015 and 2018. Educational Technology Research and Development, 68(3), 853–876. https://doi.org/10.1007/s11423-019-09700-4
• *Hilton, J., III., Fischer, L., Wiley, D., & Williams, L. (2016). Maintaining momentum toward graduation: OER and the course throughput rate. International Review of Research in Open and Distributed Learning, 17(6), 18–27. https://doi.org/10.19173/irrodl.v17i6.2686
• *Hilton, J. L., III., Gaudet, D., Clark, P., Robinson, J., & Wiley, D. (2013). The adoption of open educational resources by one community college math department. International Review of Research in Open and Distributed Learning, 14(4), 37–50. https://doi.org/10.19173/irrodl.v14i4.1523
• Huang, R., Tlili, A., Chang, T. W., Zhang, X., Nascimbeni, F., & Burgos, D. (2020). Disrupted classes, undisrupted learning during COVID-19 outbreak in China: application of open educational practices and resources. Smart Learning Environments, 7, 1-15. https://doi.org/10.1186/s40561-020-00125-8
• Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383(9912), 166–175. https://doi.org/10.1016/S0140-6736(13)62227-8
• *Jhangiani, R. S., Dastur, F. N., Le Grand, R., & Penner, K. (2018). As good or better than commercial textbooks: Students’perceptions and outcomes from using open digital and open print textbooks. Canadian Journal for the Scholarship of Teaching and Learning. https://doi.org/10.5206/cjsotl-rcacea.2018.1.5
• Kahle, D. (2008). Designing open educational technology. In T. Iiyoshi and M. S. Vijay Kumar (Eds.), Opening up education: The collective advancement of education through open technology, open content, and open knowledge (pp.27–45). MIT Press. https://mitpress.mit.edu/9780262515016/opening-up-education/
• *Kelly, D. P., & Rutherford, T. (2017). Khan Academy as supplemental instruction: A controlled study of a computer-based mathematics intervention. The International Review of Research in Open and Distributed Learning. https://doi.org/10.19173/irrodl.v18i4.2984
• Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. (EBSE 2007–001). Keele University and Durham University Joint Report.
• Konstantopoulos, S. P. Y. R. O. S., & Hedges, L. V. (2019). Statistically analyzing efect sizes: Fixed-and random-efects models. The Handbook of Research Synthesis and Meta-Analysis (pp. 245–280). Russell Sage Foundation. https://doi.org/10.7758/9781610448864.15
• *Lawrence, C. N., & Lester, J. A. (2018). Evaluating the efectiveness of adopting open educational resources in an introductory American government course. Journal of Political Science Education, 14(4), 555–566. https://doi.org/10.1080/15512169.2017.1422739
• Liesa-Orus, M., Lozano Blasco, R., & Arce-Romeral, L. (2023). Digital Competence in University Lecturers: A Meta-Analysis of Teaching Challenges. Education Sciences, 13(5), 508. https://doi.org/10.3390/educsci13050508
• *Medley-Rath, S. (2018). Does the type of textbook matter? Results of a study of free electronic reading materials at a community college. Community College Journal of Research and Practice, 42(12), 908–918. https://doi.org/10.1080/10668926.2017.1389316
• Morris, S. B. (2008). Estimating efect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364–386.
• OLCOS. (2007). Open Educational Practices and Resources. Available online: https://www.olcos.org/cms/upload/docs/olcos_roadmap.pdf
• Orwin, R. G. (1983). A fail-safe N for efect size in meta-analysis. Journal of Educational Statistics, 8(2), 157–159. https://doi.org/10.3102/10769986008002157
• Otto, D., Schroeder, N., Diekmann, D., & Sander, P. (2021). Trends and gaps in empirical research on open educational resources (OER): A systematic mapping of the literature from 2015 to 2019. Contemporary Educational Technology,13(4), ep325. https://doi.org/10.30935/cedtech/11145
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hofmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaf, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, https://doi.org/10.1136/bmj.n71
• Pawlyshyn, N., Braddlee, D., Casper, L., & Miller, H. (2013). Adopting OER: A case study of cross-institutional collaboration and innovation. Educause Review. Accessed on May 20, 2023 from https://er.educause.edu/articles/2013/11/adopting-oer-a-case-study-of-crossinstitutional-collaboration-and-innovation.
• *Robinson, T. J. (2015). The efects of open educational resource adoption on measures of post-secondary student success. Brigham Young University.
• Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literatura reviews. Annual Review of Psychology, 52(1), 59–82. https://doi.org/10.1146/annurev.psych.52.1.59
• Salomon, G., & Clark, R. (1977). Reexamining the methodology of research on media and technology in education. Review of Educational Research, 47(1), 99–120. https://doi.org/10.3102/00346543047001099
• Shear, L., Means, B., & Lundh, P. (2015). Research on open: OER research hub review and futures for research on OER. SRI International: Menlo Park, CA, USA.
• *Shemy, N., & Al-Habsi, M. (2021). The efect of a Training Program based on Open Educational Resources on the Teachers Online Professional Development and their Attitudes towards it of AL-Dakhliya Governorate in Sultanate of Oman.Journal of Elearning and Knowledge Society, 17(1), 18–28. https://doi.org/10.20368/1971-8829/1135283
• Shi, W., Ghisi, G. L. M., Zhang, L., Hyun, K., Pakosh, M., & Gallagher, R. (2023). Systematic review, meta-analysis and metaregression to determine the efects of patient education on health behaviour change in adults diagnosed with coronary heart disease. Journal of Clinical Nursing, 32(15–16), 5300–5327. https://doi.org/10.1111/jocn.16519
• Slavin, R. E., & Lake, C. (2008). Efective programs in elementary mathematics: A best-evidence synthesis. Review of Educational Research, 78(3), 427-515. https://doi.org/10.3102/0034654308317473
• Smith, M. (2013). Ruminations on Research on Open Educational Resources. William and Flora Hewlett Foundation. Retrieved from https://hewlett.org/library/ruminations-on-research-on-open-educational-resources/
• *Sulisworo, D., & Basriyah, K. (2021). Problem based learning using open educational resources to enhance higher order thinking skills in physics learning. In Journal of Physics: Conference Series, 1783(1), 012108. https://doi.org/10.1088/1742-6596/1783/1/012108
• Thalheimer, W., & Cook, S. (2002). How to calculate efect sizes from published research: A simplifed methodology. WorkLearning Research, 1(9).
• Thomas, L., & Krebs, C. J. (1997). A review of statistical power analysis software. Bulletin of the Ecological Society of America, 78(2), 126–138.
• Thorlund, K., & Mills, E. J. (2012). Sample size and power considerations in network meta-analysis.Systematic Reviews, 1, 1-13. https://doi.org/10.1186/2046-4053-1-41
• Ting, K. M. (2010). Precision and Recall BT—Encyclopedia of Machine Learning (C. Sammut & G. I. Webb (Hrsg.); S. 781). Springer US. https://doi.org/10.1007/978-0-387-30164-8_652
• Tlili, A., Altinay, F., Huang, R., Altinay, Z., Olivier, J., Mishra, S., Jemni, M., & Burgos, D. (2022). Are we there yet? A systematic literature review of Open Educational Resources in Africa: A combined content and bibliometric analysis. PLoS ONE, 17(1), e0262615. https://doi.org/10.1371/journal.pone.0262615
• Tlili, A., & Burgos, D. (2022). Unleashing the power of Open Educational Practices (OEP) through Artifcial Intelligence (AI): Where to begin? Interactive Learning Environments. https://doi.org/10.1080/10494820.2022.2101595
• Tlili, A., Huang, R., Chang, T. W., Nascimbeni, F., & Burgos, D. (2019). Open educational resources and practices in China: A systematic literature review. Sustainability, 11(18), 4867. https://doi.org/10.3390/su11184867
• Tlili, A., Jemni, M., Khribi, M. K., Huang, R., Chang, T. W., & Liu, D. (2020). Current state of open educational resources in the Arab region: An investigation in 22 countries. Smart Learning Environments, 7, 1–15. https://doi.org/10.1186/s40561-020-00120-z
• Tlili, A., Zhang, J., Papamitsiou, Z., Manske, S., Huang, R., Kinshuk, & Hoppe, H. U. (2021). Towards utilising emerging technologies to address the challenges of using Open Educational Resources: a vision of the future. Educational Technology Research and Development, 69, 515-532. https://doi.org/10.1007/s11423-021-09993-4
• UN. (2021). Sustainable Development Goals. United Nations. https://www.un.org/sustainabledevelopment/
• UNESCO. (2019). Recommendation on Open Educational Resources. UNESCO: Paris, France.Accessible from: https://www.unesco.org/en/legal-afairs/recommendation-open-educational-resources-oer
• Vankov, I., Bowers, J., & Munafò, M. R. (2014). Article commentary: On the persistence of low power in psychological science. Quarterly Journal of Experimental Psychology, 67(5), 1037–1040. https://doi.org/10.1080/17470218.2014.885986
• Wang, H., Tlili, A., Huang, R., Cai, Z., Li, M., Cheng, Z., Yang, D., Li, M., Zhu, X., & Fei, C. (2023). Examining the applications of intelligent tutoring systems in real educational contexts: A systematic literature review from the social experiment perspective. Education and Information Technologies. https://doi.org/10.1007/s10639-022-11555-x
• Weller, M. (2010). Big and Little OER. Open Ed, Barcelona. http://hdl.handle.net/10609/4851
• Weller, M. (2017). The Development of New Disciplines in Education – the Open Education Example. https://oro.open.ac.uk/49737/
• Weller, M., de los Arcos, B., Farrow, R., Pitt, B., & McAndrew, P. (2015). The Impact of OER on Teaching and Learning Practice. Open Praxis, 7(4), 351–361. https://doi.org/10.5944/openpraxis.7.4.227
• *Westermann Juárez, W., & Venegas Muggli, J. I. (2017). Efectiveness of OER use in frstyear higher education students’mathematical course performance: A case study. In C. Hodgkinson-Williams & P. B. Arinto (Eds.), Adoption and impact of OER in the Global South (pp. 187–229). https://doi.org/10.5281/zenodo.601203
• Wiley, D. (2014). The Access Compromise and the 5th R [blog post]. Iterating toward openness. Improving Learning: Eclectic, Pragmatic, Enthusiastic. https://opencontent.org/blog/archives/3221
• Wiley, D. (2022). On the Relationship Between Adopting OER and Improving Student Outcomes. https://opencontent.org/blog/archives/6949
• Wiley, D., & Hilton, J. L., III. (2018). Defning OER-enabled pedagogy. International Review of Research in Open and Distributed Learning. https://doi.org/10.19173/irrodl.v19i4.3601
• *Winitzky-Stephens, J. R., & Pickavance, J. (2017). Open educational resources and student course outcomes: A multilevel analysis. International Review of Research in Open and Distributed Learning, 18(4), 35–49. https://doi.org/10.19173/irrodl.v18i4.3118
• Yuan, M., & Recker, M. (2015). Not all rubrics are equal: A review of rubrics for evaluating the quality of open educational resources. International Review of Research in Open and Distributed Learning, 16(5), 16–38. https://doi.org/10.19173/irrodl.v16i5.2389
• Zhang, X., Tlili, A., Huang, R., Chang, T., Burgos, D., Yang, J., & Zhang, J. (2020b). A case study of applying open educational practices in higher education during COVID-19: Impacts on learning motivation and perceptions. Sustainability, 12(21), 9129. https://doi.org/10.3390/su12219129
• Zhang, X., Tlili, A., Nascimbeni, F., Burgos, D., Huang, R., Chang, T. W., Jemni, M., & Khribi, M. K. (2020a). Accessibility within open educational resources and practices for disabled learners: a systematic literature review. Smart Learning Environments, 7, 1–19. https://doi.org/10.1186/s40561-019-0113-2
• Zulaiha, D., & Triana, Y. (2023). Students’ perception toward the use of open educational resources to improve writing skills. Studies in English Language and Education, 10(1), 174–196. https://doi.org/10.24815/siele.v10i1.25797