Clinical rotations in hospitals are an integral part of the medical school curricula and are a highly influential component of student learning. Clinical rotations also provide students with practical skills such as history taking, physical examination, and clinical reasoning.1 Clinical rotations have been shown to enhance student development by improving their knowledge and skills as well as leading to other areas of development, which are arguably unachievable in a classroom context such as developing relationships, forming a sense of professional identity, and modifying attitudes.1–4

The quality of supervision provided by clinical tutors at these rotations is essential for the effectiveness of the teaching.2,4 However, clinical staff nowadays experience increased demands, and this has been found to diminish the quality of clinical education.5 Therefore, providing an instrument to measure clinical tutor effectiveness at clinical rotations could be a beneficial tool to understand and enhance clinical supervision. The Maastricht Clinical Teaching Questionnaire (MCTQ) was developed by Stalmeijer et al,6 to evaluate clinical teachers’ supervisory skills during undergraduate clinical rotations in the medical curriculum.

The MCTQ is based on the cognitive apprenticeship model, which emphasizes teachers’ cognitive processes while performing a complex task. It is made up of six teaching strategies: modeling, coaching, scaffolding, articulation, reflection, and exploration. Researchers have demonstrated effectiveness in implementing a cognitive apprenticeship model of instruction in educational settings.7,8 The first teaching strategy ‘modeling’ occurs in two parts. Behavioral modeling is when a person observes an experienced individual giving instructions. The second part is cognitive modeling, and that is when the instructor expresses their thought process. Coaching is when an instructor observes a student while he/she is performing a task and provides feedback. In scaffolding, the student demonstrates some form of mastery over the concept. In this stage, the instructor provides selective feedback. The fourth teaching method ‘articulation’ is when the students articulate their understanding of a specific task using an assessment of content mastery. Reflection occurs when the students are allowed to reflect on their understanding of the concept and strategies in problem-solving and compare them to other students or experts. Lastly, exploration is when the students discuss what they have learned and can understand how the information they learned can be applied.

The MCTQ has been validated and shown to be beneficial for evaluating clinical teachers during rotations.5,9 This instrument could help improve the quality of education by identifying the areas of improvement where more training is needed and can also help faculty developments measure the return of their investment in training these identified areas.10 Moreover, measuring teaching effectiveness is also beneficial in guiding, supporting, and motivating the clinical tutors to improve their teaching.11–13

The validity of this questionnaire has been established in different types of medical contexts (general and veterinary).5,9 Additionally, the validity of this instrument has been demonstrated in different countries including the Netherlands, Australia, Canada, Columbia, Ireland, and the UK.5,14–18 However, this study has not been validated in a Middle Eastern context such as Bahrain. Educational practices are context-specific; therefore, culture is a key component that can influence validity in instruments that evaluate the quality of teaching.19 Demonstrating validity of this questionnaire across many different cultures will strengthen its use as well as demonstrate the generalizability of the questionnaire across different cultural contexts. This will also allow more people to use and benefit from this instrument.

Our study sought to evaluate the psychometric properties of the MCTQ in medical students in Bahrain. This will be achieved by assessing the construct validity, internal reliability, performing factor analysis, conducting a generalizability study, and measuring the relationship between instrument scores and other variables with relevance to the construct being measured.

Methods

This study took place at the Arabian Gulf University (AGU) in Bahrain between 2016 and 2017. Questionnaires were distributed to all medical students who were asked to evaluate 98 clinical tutors at the university. A total of 549 questionnaire responses were collected.

Data was collected using the MCTQ.6 The questionnaire is composed of 24 items, which involve a set of statements scored on a five-point Likert scale from fully disagree to fully agree. The second section of the questionnaire asks participants to provide an overall assessment of their tutor by rating their skills on a scale ranging from 1 to 10. A higher score indicates a better overall assessment.

Each item in the questionnaire belongs to one of six domains. The first domain is the general learning climate. This domain involves statements that intend to measure the extent to which the tutor creates a safe learning environment for the students and treats them with respect. The second domain is modeling, which assesses whether tutors carry out a task by acting as a role model and create an opportunity for students to observe and build a conceptual model of the process necessary to complete the task. The coaching domain measures whether the tutor observes students while performing different tasks and gives feedback. The articulation domain involves statements that measure whether the tutors ask students to explain their actions for them to be aware of gaps in their knowledge and skills to increase their understanding and motivate them to ask questions. The fifth domain is reflection, and it assesses whether the tutor encourages students to be aware of their strengths and weaknesses and consider what they can do to improve things. The final domain is exploration, which assesses whether the tutors encourage the students to formulate learning objectives based on identifying strengths and weaknesses and challenging the students to learn new things.

Exploratory factor analysis (EFA) was used to assess validity, and the analysis was conducted using SPSS Statistics (SPSS Inc. Released 2007. SPSS for Windows, Version 16.0. Chicago, SPSS Inc). A Kaiser-Meyer-Olkin (KMO) analysis and Bartlett’s test of sphericity were conducted to determine if the data was suitable to continue with principal component analysis.

Confirmatory factor analysis (CFA) was used to determine the construct validity of the MCTQ. IBM SPSS AMOS V.22 for Windows was used for this analysis. First, the normality of the distribution was assessed by calculating the skewness and kurtosis values of all the data. The data was normally distributed. The estimation method for CFA was maximum likelihood estimation. Several fit indices were used to evaluate the model fit. The fit indices taken into account were relative chi-square divided by degrees of freedom (CMIN/df), goodness of fit index (GFI), comparative fit index (CFI), root mean square error of approximation (RMSEA), non-normalized fit index (NNFI), normalized fit index (NFI), and standardized root mean square residual (SRMR).20,21 A CMIN/df value < 2 indicated a good fit.17 The GFI, CFI, NFI, and NNFI values range from 0–1, values≥ 0.80 indicate an acceptable model fit.22,23 An RMSEA value between 0.08 and 0.10 suggests an average model of fit and a value < 0.08 demonstrates a good fit. SRMR values < 0.08 demonstrate a good fit.24

Inter-rater reliability was assessed using a generalizability study to determine the number of student ratings required to provide reliable feedback for teachers using the factors derived from the CFA. For this analysis, uRGeneva (G-String-IV) version 6.3.8 was used. A generalizability (G) coefficient of at least 0.70 was required to demonstrate good reliability.

Moreover, Cronbach’s alphas were computed for each scale to determine internal consistency reliability. Coefficients > 0.70 were considered acceptable. This analysis was conducted using SPSS.

The AGU research ethical committee approved the study, which was in accordance with the Declaration of Helsinki and comparable ethical standards. All participants were assured that participation was voluntary and that their data would remain anonymous and may be published.

Results

Table 1 includes the average score responses for each item of the questionnaire including the overall tutor assessment. To assess the validity of MCTQ, an EFA was conducted. The EFA revealed a KMO value of 0.976, and Bartlett’s test of sphericity was significant (p < 0.001). The results were sufficient enough to allow the analysis to continue with a principal component approach.

The item-loadings were first observed in the sixth-factor model. However, in the sixth-factor model, only two items loaded on the fifth component (minimum items per component is three). The next step was to examine the factor loadings in a five-factor model. This yielded unsatisfactory results as well as no items loaded on the fifth factor. In the four-factor alternative, each factor had more than three items, and all values had a high loading (> 0.45). Thus, a four-factor model with 24 items was more appropriate for this study. The total four factors explained an overall 87.1% of the total variance [Table 2].

The CFA revealed that the original 24-item model with six factors did not fit the data [Table 3]. After reducing the number of factors and reorganizing items according to the modification indices, an acceptable model of fit was found for a 24-item questionnaire with four factors. The following results were obtained for the four-factor model: CMIN/df was 5.026, CFI, GFI, NFI, and NNFI were all > 0.80 (0.955, 0.858, 0.950, and 0.952, respectively), SRMR was 0.016 and the RMSEA score was 0.086. The results show that all indices met the criteria for a reasonable model of fit, except for CMIN/df. Moreover, the correlations between the factors and with overall tutor assessment varied between 0.874 and 0.930.

The results from the CFA are presented in Figure 1. The standardized solution is given, and parameters estimates for latent factors are shown. The lambda-ksi estimates are analogous of factors loadings in EFA, and values of 0.79 or higher for these parameters indicate well-defined latent constructs. The lambda-ksi estimates range from 0.74–0.95. Learning environment and modeling demonstrate the greatest variability in the magnitude of their estimates with values ranging from 0.74–0.92 and 0.79–0.92, respectively. This variability corresponds to their lower internal reliability coefficients relative to the other teaching scales coefficient alphas are 0.962 and 0.960, respectively.