Fecha de recepción: 18/04/2023 / Fecha de revisión: 19/09/2023 / Fecha de aceptación: 01/10/2023

Introduction

This article arises within the framework of the European LOVEDISTANCE project, which was funded under the promise of compliance with certain standards and impacts in the educational field, and a very important one has to do with the quality of distance education received by the target groups, so two possible options were considered:

• Create from scratch an instrument for EVA assessment to be used in the LOVEDISTANCE project.
• Use an instrument already validated for the assessment of VAS, which will be used in the LOVEDISTANCE project.

We opted for the latter, i.e., the use of a ready-made and validated instrument due to the particularities of the LOVEDISTANCE project with respect to its Inclusive Education approach and the wide variety of countries and regions targeted, but also as a matter of educational pragmatism in which, through the literature, it was discovered that there were already good precedents. For example, the DELES, developed by Walker and Fraser (2005), which is structured with 34 items in 6 scales; and the WEBLEI, developed by Chang and Fisher (2003), which is structured with 32 items in 4 scales.

The use of both has been validated in multiple researches (Valencia et al., 2014), and although they are among the most used and recognized as useful instruments for the assessment of EVA in the educational dimension pointed out by Salinas (2011), or the one referred to Educational Quality pointed out by Torres and Ortega (2003), the DELES was chosen because it has a broader and more developed scale to assess student attention as shown in Figure 1, which is very important in the LOVEDISTANCE project originally proposed, however, the problems posed by the use of the DELES were synthesized in two aspects:

• Lack of updating: the DELES was created and validated in 2005, since then there have been changes and advances in educational and technological aspects
• The lack of an approach towards inclusive education: and it is believed necessary that EVA has a special orientation towards it.

Figure 1
Instructor support on DELES Scale 1

Note. The figure represents the first evaluation scale of the DELES, the one on which special emphasis is placed for the purposes of the LOVEDISTANCE project.

Therefore, the objective of this research is to revalidate the DELES for its possible use in the LOVEDISTANCE project, and if necessary, to make the pertinent adaptations and modifications required based on the expert analysis.

The research question posed is as follows: is the EVA assessment tool, DELES, useful and valid in its current form for use in the LOVEDISTANCE project and, if modifications are required, what are they?

LOVEDISTANCE Project

The European project LOVEDISTANCE is funded by the European Union and aims to promote inclusive education in Israel and Georgia by expanding access to higher education for potential and existing students from vulnerable groups, religious and ethnic minorities, refugees, working students and students living in peripheral/distant/rural areas. There are different concepts around the idea of inclusive education, but the one that comes closest to the one proposed by the LOVEDISTANCE project is that of Clavijo and Bautista-Cerro (2020), where they refer that inclusion in the educational environment entails attitudes of deep respect for differences and a responsibility to make them an opportunity for development, participation and learning. The right to education is an unquestionable human right in any modern society. It was established in Art. 26 of the Universal Declaration of Human Rights and developed in many subsequent documents, such as the United Nations International Covenant on Economic, Social and Cultural Rights in 1976, the Convention on the Rights of the Child in 1990, the Millennium Development Goals in 2000 or the Sustainable Development Goals of 2014, among many others.

The main objectives of the LOVEDISTANCE Project are:

• Build capacity in Israel and Georgia to enable their higher education systems to adapt to effective distance learning programs. These capabilities must be related to institutional, staff and student preparedness, as well as to closing knowledge gaps.
• Develop curricula in Israel and Georgia based on distance learning that meet the equity and accessibility requirements of the partner countries' higher education systems to improve the educational integration of disadvantaged students into their educational system (identified target groups).
• Facilitate the accessibility of transfer to teaching and learning materials in electronic format for students in the identified target groups.
• Train and coach faculty members, professional and administrative staff in the design, development, implementation and quality assessment of distance education and e/b-learning courses.
• Raise public awareness about access, equity and democratization of HE to promote social inclusion.

The project aims to provide a holistic view and a solution to develop the total capacity required for the transition from the traditional, frontal model of teaching in higher education institutes to distance learning. Quality standards in higher education now demand the integration of technology into their teaching methods, whether face-to-face, inverted, hybrid or online-only. Special emphasis is placed on pedagogical, technological and educational know-how.

With this set of tools, the project aims to provide the basic skills and competencies that virtual teachers should master, specifically for the target populations, and to promote a training system to achieve this goal.

The project included the design of the framework and infographics for an online course. It can be used by teachers who are developing distance courses with digital contents and resources, using different communication and evaluation tools.

Virtual learning environments (VLE): Concept, characteristics and evaluation

Having clarified the objective and approach of the LOVEDISTANCE project, it is appropriate to discuss EVA and the instrument to be used to assess virtual learning environments, through which inclusive education is delivered to all target groups of the project.

According to Cedeño (2019), an EVA is an educational space hosted on the web, made up of a set of computer tools that enable didactic interaction, which is becoming more and more valid and relevant, but its use was enhanced with the COVID-19 pandemic.

For Belloch (2013), VLEs involve a combination of resources, interactivity, support and structured learning activities, and their main characteristics are:

• Interactivity: the teacher or tutor should not be the protagonist but, on the contrary, the learner should be the main actor
• Flexibility: understood as the set of functionalities that allow the whole system to be easily adapted to the organization where it will be implemented
• Scalability: ability to operate optimally with a small or large number of users
• Standardization: possibility of importing and exporting content in standard formats

This combination of resources implies the implementation of human, pedagogical, technical and technological resources for the optimal functioning of an EVA. In accordance with this idea, Salinas (2011) defines an EVA as having four basic characteristics:

• It is an electronic environment, not material in a physical sense
• It is hosted on the web and can be accessed via the Internet
• Technological support and technical support for troubleshooting are available
• The didactic relationship is not face-to-face

These four characteristics frame two important dimensions in the EVA: the technological and the educational, which are interrelated and enhance each other.

The technological dimension is represented by the tools or computer applications with which the environment is built. These tools serve as support or infrastructure for the development of educational proposals.

The educational dimension of an EVA is represented by the teaching and learning process that takes place within it. This dimension indicates that it is a human and social space, essentially dynamic, based on the interaction generated between the teacher and the students from the planning and resolution of didactic activities. An EVA is presented as an environment to promote learning based on multidirectional communication processes (teacher/student - student/teacher and students among themselves). It is a shared work environment for the construction of knowledge based on the active participation and cooperation of all members of the group.

In relation to the educational dimension, VAS have had a positive impact on the development of students' competencies (Bruffee; however, it is necessary to strengthen the interaction with students and the content feedback process (Romero and Moreira, 2020), therefore the process of continuous improvement, updating and evaluation becomes indispensable.

In order to carry out this process of continuous improvement, updating and evaluation of EVA, Torres and Ortega (2003) propose four areas of analysis:

• Technical quality: referred to the technical characteristics of the platform. It is mostly related to the technological infrastructure, the cost of access and maintenance, the knowledge required for its use, the ease of navigation through the interface, the quality of the security control systems, the versatility for monitoring registrations, cancellations and other eventualities.
• Organizational and creative quality: these are the organizational potentialities for the optimal functioning of the Teaching-Learning process. It is related to flexibility when giving instructions, adaptation to other educational environments, versatility to design and implement help systems for students, availability of design tools, possibility of organizing content at convenience and multimedia integration.
• Communicational quality: the possibility of synchronous and asynchronous communication both with students and with others involved in the Teaching-Learning process. Applies to discussion groups, messaging, message notification, calendaring and conferences
• Didactic quality: possibility of integrating different training strategies that allow the achievement of learning objectives, following the principles of order and clarity, autonomy, active, meaningful and cooperative learning.

Some experiences, such as those of Valencia et al. (2014) are interesting for evaluating VAS in the areas of analysis described by Torres and Ortega (2003), where the design of instruments such as "compliance rubrics" and validation with Likert-type questionnaires is the most widely used quantitative-qualitative option, from the pedagogical perspective, to measure the functionality and performance of the different educational elements that compose them, and that VAS should desirably have. Regarding the subject of rubrics and Likert-type questionnaires, Gottlieb (2006) mentions that they are ideal tools for the evaluation of instruments or techniques used in the educational field, since they represent a scoring guide with specified criteria used to interpret performance objectively, and whose use facilitates correction and feedback (Carrasco, 2007).

In this sense, Cano (2015) conducts a review of Likert-type rubrics and questionnaires as assessment resources in higher education, associating them as an assessment tool according to a vision of competencies, which is the current paradigm of Western educational models, thus concluding that the best way to assess VAS in a general sense are Likert-type rubrics and questionnaires compared to other instruments.

DELES

The Distance Education Learning Environments Survey (DELES) was developed by Walker and Fraser in 2005. The DELES, as shown in Figure 2, has 4 items broken down into 4 scales: (1) Instructor Support; (2) Student Collaboration and Interaction; (3) Personal Relevance; (4) Authentic Learning; (5) Active Learning; and (6) Student Autonomy. The DELES is an online instrument that can be used by students anywhere, eliminates data transfer errors and does not allow for non-response, which increases the overall validity of the instrument. The development of DELES relied heavily on high-quality distance education literature and expert content validation techniques. It treats distance learning as a social and psychological climate distinct from that found in other post-secondary face-to-face settings. Regarding the validity of the DELES, according to analyses of data from a sample of 680 students, the DELES showed strong factorial validity and internal consistency reliability.

Figure 2

Scales and items that make up the DELES

Method

Empirical data collection

The research exercise described in this article was conducted within the framework of a transnational working meeting at Levinsky University, Tel Aviv, in April 2022.

The organization of the academic exercise that resulted in modifications to the instruments for evaluating EVA was as follows:

Three teams of 9 members each were formed. The objective of this organization was, on the one hand, to divide the work into smaller and more functional teams, and on the other hand, to form groups in a random and collaborative way. The participants were education experts, leaders and academics from 3 different countries: 13 from Israel, 3 from Portugal and 11 from Georgia. The profile of each of them could belong to one of the following groups:

1. Professor-Researcher expert in Information and Communication Technologies in the educational area
2. Professor-Researcher expert in Higher Education
3. Professor-Researcher expert in Inclusive Education

Likert scale questionnaires were distributed with the following initial assessment criteria.

1. Correction level 1: the item does not require any modification and is in perfect agreement with the scale and the instrument in general
2. Level 2 correctness: the item can improve the wording and syntax, but is relevant within the scale and with the instrument in general
3. Correction level 3: the item must change the sense in which the idea is stated, but it has relevance within the scale and with the instrument in general
4. Correction level 4: the item must be replaced by another, and its relevance within the scale and the instrument in general is questioned
5. Correction level 5: the item should be deleted, should not be replaced and has no relevance.

At the end of the questionnaire, a comments section was also included to make criticisms or suggestions to the instrument in a more qualitative way.

Information processing

The measure of consensus among experts was used, which is defined as consensus as an opinion or position reached by a group of people as general agreement (Tastle & Wierman, 2007). Cronbach's alpha was also used to measure the reliability of the empirical data collection, that is, to assess the extent to which the items of the DELES instrument are correlated. And finally, an ANOVA analysis was performed to see if there were differences between the three groups.

As shown in the equation, consensus is a measure of attraction to a mean value:

where:

• X= list of categories ("1. Insignificant (I)" ... "5. The Most Significant (TMS)").
• p_i= probability of each X.
• d_x= X_max-X_min.
• X_i=particular element of X.
• = mean or expected value.

It is, therefore, a measure of dispersion for ordinal data in the interval [0 1] and which, on a Likert scale with gradation between responses, can be transformed into the form of percentage of agreement, as shown in Table 1.

Table 1

Expert Consensus Interpretation

Interval	Consensus classification
	Very strong consensus
	Strong consensus
	Moderate consensus
	Balance
	Moderate dissent
	Strong dissent
	Very strong dissent

Note. Adapted from Wierman & Tastle (2005)

In relation to Cronbach's Alpha, the equation is as follows:

where:

• K= number of items
• σ²_Yi = variance of item i
• σ²_X = variance of the observed scores of the individuals.

The value of Alpha can assume values between 0 and 1. Values close to 1 are better, since they indicate greater internal consistency. By convention and for practical purposes, Alpha values equal to or greater than 0.6 are considered acceptable, greater than 0.8 are good, and greater than 0.9 are excellent. Values below 0.5 and close to 0 indicate that a scale has poor reliability.

Results

Consensus

The application of the consensus formula resulted in the consensuses shown in Table 2. As can be seen, a moderate consensus was obtained in the three groups of experts.

Table 2

Results of the expert consensus

Item	Consensus (%)
Group 1	62 (Moderate)
Group 2	62.5 (Moderate)
Group 3	60.46 (Moderate)
Media	61.66
SD	25.63

Cronbach's Alpha

Result= 0.60

Reliability of the instrument: moderate

Differences between groups of experts

The sum of the individual item ratings for each group of experts is shown in Table 3.

Table 3

Sum of individual ratings for each group of experts

GROUP 1	GROUP 2	GROUP 3
81	76	76
63	81	91
86	78	77
68	79	76
73	72	85
66	67	67
70	88	57
67	78	69
59	84	76

Table ANOVA

The Analysis of Variance resulted in the values shown in Table 4.

Table 4

Analysis of variance table

F.V.	SS	df	MS	F	Sig.
Between	274.888	2	137.444	1.966	0.1619
Within	1677.77	24	69.907
Total	1952.667	26

Given that 0.1619 >> p=0.05 it is concluded that there are no significant disparities between the groups' means, so the differences between them are attributed to chance.

Discussion and Conclusions

The purpose of this research was to evaluate the use of the DELES instrument for the evaluation of EVA in the distance education project with a focus on inclusion of vulnerable groups called LOVEDISTANCE, as well as to consider its possible redesign and updating. The main findings of this study will be discussed below.

From the results obtained in this research, it can be deduced that the DELES is a suitable evaluation instrument for the LOVEDISTANCE project in its original approach, since the experts' evaluation ratifies that, statistically, it is a useful, valid and pertinent tool; which answers the research question and fulfills the general objective; however, there are qualitative nuances in this interpretation that, next, will be broken down scale by scale.

From the analysis of the results obtained in relation to the first scale of the DELES, i.e., that related to "Instructor Support", it can be affirmed that it is still convenient to simplify the scale, on the one hand; and on the other, to make an evaluation with a more positive orientation. Regarding positive evaluation, it is believed to be more useful than other types of evaluation, in addition to being more in line with new educational trends and positive psychology (Escudero et al., 2008) but also with the heart of the LOVEDISTANCE Project, whose emphasis lies in accessibility. At this point, some researchers suggested the identification of opportunities for improvement more in the sense of an evaluation by competencies and not so much as a simple signaling of the good and the bad that is being done in the EVA, and in this regard it is relevant what Tobón and Posada (2008) pointed out, which refers that competency-based evaluation is a process of measurement, monitoring and permanent adjustment of the educational teaching process, not only an exchange of performance indicators, where the communicative and qualitative part is also very relevant, especially in the context of excluded groups.

The other point on which there was consensus among researchers on the first DELES scale was the need to simplify it, because it is not necessary to have so many items to determine whether an instructor is doing his or her job correctly or not. Difficulties in obtaining instructor involvement occur regularly and have resulted in different approaches to instructor support (Bianco et al., 2002). Simplification tends to eliminate duplication or confusion in the evaluation (Bruffee, 1993), and it is also a primary requirement to improve the quality of an instrument when it is being readapted, as is the case of the DELES.

Regarding the second scale of the DELES, i.e., that related to "student interaction and collaboration", the results suggest a simplification of this section in the instrument, and at the same time, to give a more important relative weight to maintaining a constant interaction with marginalized groups in the educational context. The virtual teacher has been characterized by creating new learning practices, where knowledge emerges through interaction, accompaniment, as well as the feedback of activities and resources integrated in virtual training for the achievement of the proposed objectives, that is, in virtual learning environments, the virtual teacher is part of an interdisciplinary team that contributes to the development of environments in accordance with the demands of the knowledge and information society (Coll & Monereo, 2008). According to Bruffe (1993), collaboration occurs when students work together in groups to create knowledge, but also work together with the teacher and transfer the nature of authority to the group. Therefore, a condition for collaboration is the teacher's ability to delegate authority and the students' ability to grant authority to each other for their own learning processes. This is an interactive process in which it is primarily the teacher's responsibility to delegate authority to the group and to promote effective interaction among group members (Forslund & Hammar, 2014).

The third scale is "personal relevance," which expresses students' interest and ability in terms of using a synchronized and asynchronized e-learning environment (Ozkok, 2020). A feature of the learning environment that emphasizes concrete and personally relevant experiences to help the learner construct individual meaning (Kwak et al., 2015). The results of this scale point to the rethinking of the items, since it is unfeasible to evaluate something that cannot be known so soon, and that is rather known in the practice of knowledge. It is not their relevance that is being questioned, but rather the formulation of the questions and their scope. Some findings made from the feedback obtained in the research exercise show that, although the study is important in the life of any person to obtain adequate and useful knowledge for individual, social and labor market needs, measuring its impact during the course of the same studies is difficult because it is not possible to appreciate a great quantitative-qualitative improvement in the short term. Therefore, it would be desirable to either simplify the scale or rethink it in terms of something more long-term.

In the next item, the instrument refers to authentic learning, and the results guide towards a simplification of the scale into one or two questions, but also highlight the importance of assessing authentic learning, which can be defined as a learning style rooted in situation cognition and problem-based learning (Ke & Kwak, 2013) that involves the learner pursuing activities that involve real or genuine information or scenarios (Kwak et al., 2015). Authentic learning typically focuses on complex real-world problems and their solutions using role-playing exercises, problem-based activities, case studies and participation in virtual communities of practice. Learning environments are inherently multidisciplinary." (Lombardi, 2007).

The findings suggest that the use of real-life case studies are the most highly rated by students and perhaps have the highest impact in terms of meaningful learning. The case study technique consists precisely in providing a series of cases that represent different real-life problem situations to be studied and analyzed. In this way, the aim is to train students in the generation of solutions (Coraggio & Vispo, 2001). Obviously, since this is an active pedagogical method, some minimum conditions are required. For example, some previous assumptions in the teacher: creativity, active methodology, concern for an integral formation, group management skills, good communication with the students and a defined teaching vocation. It must also be recognized that the method is better handled in small groups. Specifically, a case is a written account that describes a situation that occurred in the life of a person, family, group or company. Its application as a learning strategy or technique, as previously mentioned, trains students in the elaboration of valid solutions for possible complex problems that may arise in the future. In this sense, the case teaches how to live in society. And this makes it particularly important (Martínez Sánchez, 1999).

The penultimate scale of the DELES has to do with "active learning", and the results confirm that this aspect is absolutely relevant for distance education, because if it does not exist, the risk of failure is high and that of learning is null, with the difference that there can be no punctual follow-up in comparison with the face-to-face environment. Meaningful learning can be anything related to the course that all students in a class session called upon to do more than simply watch, listen, and take notes" (Felder & Brent, 2009), i.e., a learning construct that engages students to actively engage with content to construct knowledge (Prince, n.d.).

Some findings suggest that this scale could be expanded and leveled with the rest. The extension of this scale, also called "self-learning", could measure, among other things, how much the VAS facilitates the student to find the ideal space and time for study, to dedicate a fixed schedule to study and organize activities, to maintain motivation to complete the course being taken, to minimize distractions and to take some rest. Then, the challenge for this scale, according to the findings, is to modify the current items or draft new ones that go along the lines of measuring the ability of an individual to forge his own education in a self-taught manner, understood as a learning model configured by oneself to nourish oneself with all the information available to him (López, J. V. B et al., 2015)

To conclude the analysis of the scales, there remains that related to student autonomy, which can be defined as the level of student control over the planning, execution and evaluation of their own courses (Moore & Kearsley, n.d.). Autonomy in the case of distance education plays a key role as it is a key competence that, if developed, can achieve optimal student achievement, according to the results of the research. The findings show that this scale is generally positively valued and that it does not require major modifications, only some improvements or specifications regarding the possibility of the student to redirect his own learning process supported by criteria of autonomy, which, in the educational context, is intentional, conscious, explicit and analytical. Its exercise implies the determination of the learner to be responsible and to make personal decisions about his/her learning, as well as the willingness to participate, together with the teacher, in the negotiation of the following aspects: the identification of his/her own learning needs and the definition of his/her objectives; the planning of classes; the selection of contents and the establishment of their sequencing; the selection of appropriate didactic materials; the training in the use of various techniques and strategies, but especially learning and metacognitive ones (Rodríguez González, 2006).

In synthesis, the improvements to the instrument can be summarized as follows: for the first scale -instructor support-, merging questions that are repetitive and orienting it towards a more positive evaluation; for the second scale -student interaction and collaboration-, giving greater weight to contact with marginalized groups and simplifying the entire scale; in the third -personal relevance-, a rewriting of some repetitive items and orientation of others towards the long term is proposed; in the fourth -authentic learning-, rethink some items towards the deepening of learning; in the fifth -active learning- expand the scale to level it with the rest; and finally, the sixth scale -student autonomy- does not require major modifications.

Final considerations

No VAS assessment tool is definitive and immutable. In fact, most of them have multiple areas for improvement and opportunity, as demonstrated in the research exercise carried out. The initial assumption was that observations would be minimal for the DELES, however, there were significant corrections.

Although the statistical validation was correct and positive in the sense of reaffirming the hypothesis, the relevant observations and findings were obtained through the qualitative analysis of the information, which, thanks to the full willingness and collaboration of the researchers for the academic exercise, was fundamental for obtaining detailed and valuable information.

Instruments must change as VAS and educational trends evolve. Considering that the instrument in question was evaluated in 2005 -more than 15 years ago- and that technological and educational progress has been considerable since then, this would partially explain the number of observations and corrections, especially those referring to the meaning of the instrument, not to grammatical or structural ones.

The limitations of the study were, if anything, those related to time, because there was a time limit of one day for the activity according to the agenda of the meeting, and perhaps the in-depth discussion could have taken a little more time.

Although it is true that the instructions were clear regarding the execution of the dynamics and it was programmed in the initial calendar, the activity was carried out at the end of an intense day of work, which undoubtedly influenced the spirit and dedication of some researchers in this regard.

The potential number of students involved in the evaluation amounts to more than 10 thousand, and the significance of not doing so implies offering an educational service without adequate quality controls and, therefore, not being able to improve according to valid criteria.

This exercise can serve as a precedent for timely consideration for the revalidation of instruments that have already been validated, especially those that, given their characteristics, require updating and improvement. In the future, it is likely that learning environments may continue to evolve (Belloch, 2013), which is why it is not ruled out that, according to trends and updates in education and educational technology, the DELES may be re-evaluated.

As future lines of research, we visualize ideas such as validation of other instruments in the educational field or their creation and validation for the European project LOVEDISTANCE, which has an important educational challenge and the way to measure quality will be through qualitative-quantitative indicators associated with the fulfillment of previously defined goals and objectives.

Finally, it can be said that this was not a definitive or conclusive exercise, since this type of dynamics is necessary when the technological circumstances change, the educational approach or when some variable of the educational process has sufficient influence to disrupt the teaching-learning process of students in a context of social exclusion.

References

1. Cedeño, E. (2019). Entornos virtuales de aprendizaje y su rol innovador en el proceso de enseñanza. Rehuso, 4(1), 119-127. https://revistas.utm.edu.ec/index.php/Rehuso/article/view/1888
2. Coll, C. & Monereo, C. (2008) Psicología de la educación virtual. Ediciones MORATA S. L.
3. Belloch, C. (2013). Recursos tecnológicos (TIC). Unidade de tecnologia educacional (UTE) Universidade de Valência.
4. Bianco, M., Collins, B., Cooke, A., & Margaryan, A. (2002). Instructor support for new learning approaches involving technology. Staff and Educational Development International (SEDI) journal, 6(2), 129-148.
5. Carrasco, M. Á. L. (2007). Guía básica para la elaboración de rúbricas. Universidad Iberoamericana Puebla.
6. Cano, E. (2015). Las rúbricas como instrumento de evaluación de competencias en educación superior: ¿uso o abuso? Profesorado. Revista de Currículum y Formación de Profesorado, 19. http://www.redalyc.org/articulo.oa?id=56741181017
7. Chang, V., & Fisher, D. (2003). The validation and application of a new learning environment instrument for online learning in higher education. In Technology-rich learning environments: A future perspective (pp. 1-20).
8. Clavijo Castillo, R. G., & Bautista-Cerro, M. J. (2020). La educación inclusiva. Análisis y reflexiones en la educación superior ecuatoriana. Alteridad, 15(1), 113-124. https://doi.org/10.17163/alt.v15n1.2020.09
9. Coraggio y Vispo (2001) Contribución al estudio del Sistema Universitario Argentino. Miño y Dávila.
10. Escudero Muñoz, J. M., Vallejo Ruiz, M., & Botías Pelegrín, F. (2008). El asesoramiento en educación: ¿Podrían ser las competencias profesionales una contribución positiva? Profesorado: Revista de currículum y formación del profesorado, 12(1).
11. Felder, R. M., & Brent, R. (2009). Active learning: An introduction. ASQ Higher Education Brief.
12. Forslund, K., & Hammar, E. (2014). Student collaboration in group work – Inclusion as participation. University of Linköping.
13. Gottlieb, M. (2006). Assessing English language learners: Bridges from language proficiency to academic achievement. Thousand Oaks
14. Ke, F., & Kwak, D. (2013). Constructs of student-centered online learning on learning satisfaction of a diverse online student body: A structural equation modeling approach. Journal of Educational Computing Research, 48(1), 97-122.
15. Kwak, D. W., Menezes, F. M., & Sherwood, C. (2015). Assessing the impact of blended learning on student performance. Economic Record, 91(292), 91-106.
16. Lombardi, M. M. (2007). Authentic Learning for the 21st Century: An Overview. Educause Learning Initiative.
17. López, J. V. B., García, S. F., Chávez, O. R., & Porras, S. M. T. (2015)
18. Martínez Sánchez, A. (1999). El estudio de casos como técnica didáctica: Estudio de una experiencia en la enseñanza universitaria. Innovación educative, 6, 25-53.
19. Moore J. & Kearsley, 1. (s.f.). IGI Global. https://www.igi-global.com/chapter/deles-analysis-of-e-learning-environments/165778
20. Ozkok, A. (2020). Comparative Analysis of Psychosocial Dimensions of Two Higher Education Institutions’ Blended Learning Environments. Kefad - Hacettepe Üniversitesi – Turkia.
21. Prince, J. (s.f.). https://www.igi-global.com/dictionary/personal-relevance/56733
22. Rodríguez González, R. (2006). Diseño de entornos para el desarrollo de la autonomía en el aprendizaje. Aula abierta, 87, 89-103.
23. Romero, E. L. C., & Moreira, J. A. M. (2020). Entornos virtuales de aprendizaje y su rol innovador en el proceso de enseñanza. ReHuSo: Revista de Ciencias Humanísticas y Sociales, 4(1), 119-127.
24. Salinas, M. I. (2011). Entornos virtuales de aprendizaje en la escuela: tipos, modelo didáctico y rol del docente. Universidad Católica de Argentina, 12.
25. Tobón, S., & Posada, R. E. Q. (2008). Evaluación por competencias. In Primer Congreso Internacional “Competencias en la Educación del Siglo XXI.
26. Valencia Vallejo, N. G., Huertas Bustos, A. P., & Baracaldo Ramírez, P. O. (2014). Los ambientes virtuales de aprendizaje: una revisión de publicaciones entre 2003 y 2013, desde la perspectiva de la pedagogía basada en la evidencia. Revista colombiana de educación, 66, 73-103.
27. Tastle, W., & Wierman, M.J. (2007). Consensus and dissention: A Measure of Ordinal Dispersion. International Journal of Approximate Reasoning, 531-545.
28. Walker, S. L., & Fraser, B. J. (2005). Development and validation of an instrument for assessing distance education learning environments in higher education: The Distance Education Learning Environments Survey (DELES). Learning environments research, 8(3), 289-308.