Developing Instruments for Assessing the Teaching Effectiveness of Hospitality English for Hotels at Vocational Secondary Schools

This study aims to (1) develop instrument for assessing the teaching effectiveness of Hospitality English for hotels at vocational secondary schools (VSSs) in Malang, (2) examine the validity of the instruments, and (3) assess the effectiveness of the use of the instruments. This developmental study employed a modified model of Borg and Gall, consisting of four major developmental procedures: exploration, development of preliminary instrument, product field testing and revision, and final validation. The first field-testing subjects were 20 students from SMK Kota Batu and three English teachers from three VSSs. The second field-testing subjects were 26 students and two teachers from SMK Jenderal Sudirman in Malang. The third field testing subjects were 81 students and four English teachers from SMKN Kotabaru Malang dan SMK Singosari in Malang.The qualitative and quantitative data were collected using a questionnaire, interview, self-report, observation, and performance test. Alpha Cronbach’s coefficient of 0.7 was used to estimate the instrument reliability. Cohen’s Kappa was used to calculate inter-rater reliability for the observation instrument. Kane’ and Bachman & Palmer’s validation methods were used to examine the vadility of the instruments. The findings reveal process and output variables of teaching effectiveness of Hospitality English for hotels at VSSs. The process variable has 14 dimensions: planning lesson, implementing a lesson plan, establishing orientation of learning, organizing and activating knowledge, teaching ways of learning, doing varied teaching, communicating high/appropriate expectation, asking strategic questions, facilitating interaction, using realistic context, creating supportive climate, assessing communication ability, presenting subject matter, and student engagement and motivation. The inter-rater reliability analysis yielded coefficient scores of 0.77, 0.82, and 0.85 for SMKN Kotabaru, and 0.62, 0.74, 0.71 for SMK Singosari for the last three meetings.The output variable refers to skills in Hospitality English for hotels which consist of eight dimensions. The reliability analysis of the eight dimensions yielded the coefficient scores of 0.94, 0.81, 0.87, 0.95, 0.87, 0.88, 0.82, and 0.88. The affective outputs refer to satisfaction, motivation, self-confidence, and self-esteem. After unreliable items were dropped, the reliability scores of the four aspects were 0.80, 0.85, 0.85, and 0.81 respectively.Using Aiken’s coefficient, the framework devised by Kane, and the concept promoted by Bachman and Palmer, the instruments under examination are valid