In recent years, the development of virtual reality (VR) has progressed significantly.1 VR, due to its ability to design a standard, reproducible, and controllable environment,2 has been increasingly used from simple graphical application to advanced usages in the automobile industry, architecture, and medicine3 as well as the production, the visualization of science, engineering, and training.4

VR has been defined by different perspectives.5 Gaddis defined VR as: “a computer-generated simulation of the real or imagined environment or world.”6 In a definition by Sherman and Craig, the key elements to experiencing VR includes a virtual world, immersion, sensory feedback (responding to user input), and interaction.7 In other words, VR is the term used to describe a simulated 3D computer environment that allows the user to interact with this environment. Most VR environments provide tools for visual interaction with the user, although others use different types of sensors to provide a more real sense of environment.8, 9

Nowadays, VR by creating a 3D simulated environment has been considered a useful tool in many areas, such as medical education and treatment.6,10–12 VR is mainly applied in practicing medical skills, to show 3D anatomy, simulate surgery and surgical planning, and to help treat phobias, anxiety disorders, post-traumatic stress disorder, disability, and rehabilitation.6,13–17

Many research and controlled trials have shown that VR for therapeutic goals is effective, such as in phobia and anxiety disorders,18 neurorehabilitation or telerehabilitation for recovery of function after stroke and balance disorders,10,19–22 developmental delay,23 Parkinson’s disease,24 and pain management.10,25 Therefore, VR products have potential advantages for medical care and can create markets with suitable returns on investment.26

Nowadays, one of the information technology (IT) opportunities in education is simulation. For example, multi-dimensional simulation as a new method can improve medical education.27 In ‘virtual surgery’, as the most common type of medical simulations, the surgeon practices the surgery process many times and improves his/her surgical skills in a virtual environment on a virtual patient. Studies show that using virtual surgery significantly increases the quality of surgical performance in the operating room, especially in sensitive surgery such as glaucoma and brain surgery.11,28

The development and usage of VR applications need to study various dimensions of this technology. The challenges of VR are one of the most important dimensions to be determined. Despite the benefits of using technology (such as VR) in health care, there are various limitations.29–32 Also, technology tools can be part of the problem themselves.33 Lack of attention to these limitations can have serious consequences. Identifying current and potential challenges or threats help professionals to make the strategic decision about the use and develop these technologies in the healthcare sector. Our review was conducted to identify the main challenges associated with the use of VR technology in medical education and treatment.

Methods

We searched the Science Direct, Google Scholar, and PubMed databases with keywords based on three core concepts: the first one represented VR, the second challenges, and the third the context. The search string for the PubMed database is represented in Box 1. Due to the recent development of VR in medical education and treatment context, there was a risk of limited coverage of challenges; thus, the search was carried out first with the first and second core concepts, and then the context was added to the search repeated for focusing on medical context articles.

The electronic search includes all types of papers published in Persian and English languages, regardless of the publication year, up to the end of 2018. After screening the papers at the level of title and abstract, the relevant papers were selected, and the extraction of items performed. In some cases, the relevant references of articles were investigated by the author. At the final step, challenges extracted from the articles were categorized based on themes. Considerations regarding each of the challenges were also presented in the discussion section.

Results

The main challenges of the application and development of VR in medical education and treatment are described in two main general and specific categories [Figure 1].

General challenges

VR, like other information technologies, has confronted general challenges, including reduced face-to-face communications, cost limitations, educational, and users’ attitude challenges.

Reduced face-to-face communications

One of the key challenges in using telemedicine and VR applications in the health sector is its effects on face-to-face communications and medical relationships between the physician and patient.34–37 For example, in telerehabilitation, the use of VR to perform rehabilitation exercises at home or out of the healthcare organization without direct supervision or physical presence of the healthcare provider could have undesirable outcomes.32

In VR-based educational settings, because of the lack of face-to-face communication between users (such as students) and ‘real patients’, evaluations should be made in real settings to ensure the effectiveness of the program.

Education

Creating appropriate strategic guidelines based on many practical studies and their results are essential in the use of any type of new technology such as VR.3

VR technology applications in medical education and treatment, especially psychiatric treatments need a comprehensive manual that specifies how, where, and for whom this technology is appropriate. Individual characteristics (gender, age, personality, and history of motion sickness) and other unique psychological, cognitive, physical, and functional characteristics that are common in a variety of clinical conditions are very important and should be considered. Sensitivity in some patients need to be considered, including apprehensiveness to use head-mounted display (HMD), the reality test itself, learning capacity to act in a virtual environment, and the potential side effects and duration. These issues must be specified, because of ethical reasons and the effectiveness of education and treatments based on VR.3,32

Successful implementation of modern technology like VR depends on scheduling suitable educational training. In other words, the training of end-users in launching any new system is one of the success factors in the implementation and execution of programs.35

Studies show that patient age is a key factor in treatment appropriateness when using VR. For example, VR-based therapy that is suitable for children’s treatment may not be useful for older people and may provide different outcomes. Use of VR is not appropriate for the treatment of severe anxieties and may have negative effects. VR application for people that are suffering from a particular type of psychiatric illness or different psychosis properties must be done more carefully.38 Insufficient training of clinicians in the proper use of this technology for therapeutic purposes will have adverse consequences.

Besides education, preparing users according to different dimensions is very important. Neglecting user readiness confronts VR with a lot of challenges.32,36

Cost

Previous research stated that the implementation of VR is expensive.10,16,32,38 This is one of the obstacles to be met before the extensive application of the VR.16 Establishing and developing VR programs requires high-quality hardware, high-speed computers, efficient graphics cards, accurate tracking systems, high-resolution displays, and highly specialized accessories.8,32 Initial VR tools have problems, including their large size and lumpiness, the difficulty of using them, and the high cost of design and implementation, which slow down the process of using these tools.10 VR requires higher programming skills than traditional two-dimensional tools.8

Design content of VR depends on the involvement and cooperation of different professions in various fields which can make it too expensive.26,39

Users’ attitude

Users’ attitude and consequently the modern technology’s acceptance by the users will be the key non-technological factor in the utilization of technological tools by individuals.40–42 For example, some patients may be reluctant to use computers and such new technologies, which should be considered before starting treatment.32 Moreover, people who are unfamiliar with IT may also resist the use of new technologies and can not trust to use IT-based tools, especially if they do not have the required IT skills.31,38,43

Specific challenges

Designing

DESIGNING ATTRACTIVE CONTENTS

VR applications with the purpose of medical education should be designed attractively to be used more efficiently by users. Thus, in the design process, the involvement of the end-users, and engaging their views and opinions is a key factor; moreover, combining VR with augmented reality technology44 can provide more attractive content.

Patients, especially those that need to perform rehabilitation exercises, should be often encouraged to do repetitive tasks, therefore VR games designed for this purpose, should be fitted in various tastes with multiple scenarios.32 Attention to issues such as providing immediate and appropriate feedback as well as attention to the scenarios should be considered.

Several studies addressed the usability problems of the virtual environment and VR application; thus, designers should be familiar with the knowledge and methods of the human-computer
interaction field.45,46

USER-CENTERED DESIGN

User acceptance problems often lead to the abandonment of systems by users or the refusal to learn how to use them.47 The reason for user acceptance problems is the fact that systems are often designed based on technology-oriented approaches that make them difficult to learn and use.48

Designing systems, regardless of users’ comments and feedback, lead to user dissatisfaction, reduced performance, and uselessness of the system that results in the waste of human resources, as well as economic resources.49 Therefore, the need for usable and user-friendly applications is essential. In this regard, most researchers agree that user participation in the design process will help to create a usable system.50–52 Accordingly, in several studies, the involvement of end-users and their active participation in the design process has been proposed.53–55 This kind of user engagement is known as user-centered design (UCD) that combines user perspective in the application development process to achieve a well-designed and usable system.48,49,56

THE IMMERSION LEVEL AND FEELING OF PRESENCE

The main purpose of VR is to create a sense of the user’s presence in a virtual space. The created space should be believable and sufficiently interactive.11 The believability of the virtual environments depends on several factors, including the real-time processing capability of the system.57

The appropriate use of the tracking and vision systems58,59 as well as touch devices,60 which allow touching and feeling of virtual objects,1 increase the immersion level of the VR system to create a sense of presence in the virtual environment, instead of experiencing an artificial medium. Thus, the immersion level is a key moderator variable and influences the VR result for various purposes such as psychological treatments.1,32,61 As the other example, in the application of VR for pain management, researchers have demonstrated that maximizing sensory immersion and increased presence correlates with more effective pain relief.10, 25

The feeling of presence enables users that have been immersed in a virtual environment to experience emotional reactions and behavior as the real environment. However, the experiences of users may not be the same.32 The feeling of presence in the virtual environment is a complex mental mechanism that is tightly related to the emotional reasoning abilities of individuals.62 Other factors, including personal factors (e.g., concentration, emotional state, etc.) and environmental factors may affect the feeling sense. Users (patients) with depression or poor motivation for treatment may have difficulties in this regard. Paying attention to these issues and periodic monitoring may be helpful.32

In general, modeling environments for the health sector requires more than high-quality 3D objects and real-world behavior simulation of them. This requires the integration of different fields, including physics and mathematics. A user cannot have a real sense of reality when objects do not have the proper reaction so understanding the cognitive model to address this challenge is essential.26 Models and systems that aim to establish human-machine interaction in an effective, adaptable, and applicable manner must consider the cognitive aspects.26

FACILITATING INSTALLATION AND OPERATION

There are common challenges for any kind of at-home technology implementation such as logistics, learning curve, and remote technical support but some are specific to VR and implementing such systems outside the lab.31

In home-based rehabilitation VR systems, installation and operation should be as easy as possible, this process should be integrated with plug-and-play capabilities. The development of this system requires more focus on programming and eliminating conflicts between programs, driver files, complex installation processes, and calibration of motion trackers. Installing and operating the complex VR software may be challenging, especially for people with significant physical and mental disabilities or for older adults.31,32

INTERDISCIPLINARY AND MULTIDISCIPLINARY

Today’s interdisciplinary world is a basis for the progress of science. Interdisciplinary reinforcement enhances collaboration among specialists in different fields, and this leads to the creation of suitable applications for end-users and their satisfactions, as well as considering the ethical and qualitative criteria. Modeling of objects that simulates the virtual patient, the organs and equipments used in education are beyond computer science experts’ capability.26 Creating virtual worlds and characters requires specialized skills in 3D modeling, texturing, character animation, and programming.10

Safety considerations

VR TREATMENT BASED ON THE EXPERT OPINION

Like any other type of treatment, VR should only be used when it is prescribed by the appropriate clinical expert.32 Using services such as VR rehabilitation or any type of emerging technology such as telepsychology and online therapy, according to the patients’ opinion (self-diagnosis, self-help, and self-treatment) can place patients at potential risks.32,36,38

CREATING SAFETY PARAMATERS

VR can cause problems in the cognitive organizations, human experiences, memories, judgments, beliefs, and distinguishing between themselves and the environment. Having multiple virtual experiences over the years can make real judgment and self-identity difficult.38 Therefore, managing the health and safety implications of VR are important things to be considered.3 Regarding telerehabilitation, home-based VR therapy without the direct supervision of the therapist may pose certain risks.32

Most of the people may experience positive aspects of using VR, such as improvement in visualization performance or pleasure performance more than any negative effect. With the advance of technology, some of these effects may be reduced, in contrast, some may remain or even become worse. Therefore, a prospective approach is required in order to allow empirical studies to be continued and ensures that experimental data can be converted as guidance.3

Side effects of virtual reality

COMMON SIDE EFFECTS

Cybersickness and perceptuomotor after-effects have been reported as the potential side effects of VR. In fact,the main concern for VR users is simulation sickness, or cybersickness.3,10,16,32,38,63–65 Also, headaches and eye strain are seen in prolonged exposure with VR systems.31

Similar to motion sickness, cybersickness (for example, nausea, vomiting, eye fatigue, dizziness, ataxia, dizziness, etc.) is related to the conflict between different body sensory systems,32 which is due to an inconsistency between the sensory inputs. Some cybersickness cases persist such as when the real gravity is in contrast with the seen environment, for example, when the user is flying in an airplane in VR.10

The behavior of specific users (such as the consumption of alcohol and certain medications before or simultaneously with the use of HMD) may exacerbate cybersickness during VR use.32 Individual characteristics such as gender, attitude, or skill of individuals and individual psychological differences may directly affect the levels and types of symptoms and side effects of VR. Some personal characteristics are static, such as the gender and age of individuals that are unchangeable and some are dynamic such as the health status of individuals or the attitude of people about the technology, people’s VR skills and their experience of VR that the dynamic features can have transient effects.3

Since each user may experience different side effects with the use of VR, the therapist should investigate related cybersickness reports to offer suggestions to minimize the negative effects.65 Because this problem seriously threatens the development and application of VR in treatment or training programs that require frequent use of this equipment.10

Perceptuomotor after-effects that happen due to delays in the readaptation of the sensory and motor system with the real world after the virtual world (e.g., eye and hand coordination, situational instability, turbulent movements, perceptual-motor disorders, flashbacks, sleepiness, fatigue, decreased stimuli, etc.) may also be concerning. Preventing patients from some activities, such as driving and also monitoring and controlling some of their activities after using VR is recommended.32

ADDICTION TO VR-BASED TECHNOLOGIES

There is currently no evidence of the addictive nature of VR.32 However, given the fact that various studies have shown addiction to similar tools, such as games,66–68 then the precautions should be taken in this regard. This is especially true for people who are eager to use these types of programs, and it is necessary to be aware of the signs of addiction to computer programs such as excessive use of the program.68

UNNECESSARY USE AND MISUES OF VR

The current therapeutic use of VR requires a physician. However, in the future, this requirement may not be present and individuals will be able to download or purchase VR-based therapeutic and diagnostic games and applications due to the presence of available infrastructures such as smartphones and computers, through digital marketplaces.18,36,69 This potential opportunity emphasizes the need to consider multiple aspects related to the use of VR as a clinical tool or treatment modality.69

VR equipment installed on a personal computer for patients can easily be misused for other purposes. Patients should be aware of the potential consequences of using such equipment for other purposes as well as their application by other people.32

In addition, clinicians should not consider using a VR system as a replacement for a therapy program, instead, it should be used to complement traditional and usual treatment.23 In this regard, therapists should inform patients.

Evaluation and validation of VR applications

USABILITY

Another practical challenge considered by clinical researchers is the usability of VR applications.31 The existing evaluation methods in the field of system usability can be used to assure the proper design of software applications.70,71 The well-known definition of usability is provided by ISO 9241-11 (1996), that is, to what extent a system can be used by specific users to achieve specific goals with ‘effectiveness’, ‘efficacy’, and ‘satisfaction’ in a certain context.72

The ease of use and proper interface are important factors in VR-based educational applications and games. If the problems of usability are not identified and corrected by appropriate and targeted evaluations, the applications will get distance from their main goal.9,73

Most performance and usability indices lack benchmarks for VR systems.46,57 However, methods such as the Walk-through method,46 Heuristic evaluation,45 or user experience evaluation74 can help to discover some of these systems’ problems.

RESPONSIBILITY

To use modern technologies in medical treatments, it is necessary to clearly define the responsibility of the collaborating persons in the treatment process.

From an ethical perspective, patients, and therapists should be aware of the potential risks of technologies such as VR. This awareness should result in making rational decisions for its safe and effective use in the early stages of development of the program.36,69

VALIDATING PROGRAMS

Evidence showed that VR can be considered a useful tool for medical education, diagnosis, and treatment. However, this technique is still at an early stage, and it is necessary to perform controlled trials before it can be used routinely.16

Collaboration between clinical and IT experts is necessary to design VR-based training packages. This collaboration may lead to the extraction of requirements such as information structures, technical and technological components, and suitable standards.

The mere assertion that programs are efficient by companies cannot be accepted and must have empirical support.32

Discussion

The challenges of using VR technologies like any other technologies will lead to the failure or misuse of these applications. Thus, recognizing the opportunities and challenges of VR will have an important impact on the effective use of this technology in the healthcare sector.

Studies show that there are many challenges faced by using VR in medical education and treatment that have different effects. Therefore, identifying each of these factors and determining solutions should be considered.

Several studies or parts of studies have addressed the challenges of VR in a general or specific field. For example, one study addressed 10 types of VR problems, including digital modeling, complexity, model validity, model similarity, image quality, real-time performance, basic elements, materials and behavior models, image integration, and performance evaluation.57 Another review study raised key issues in the implementation of clinical research for VR that were theoretical immaturity, a lack of technical standards, the problems of separating effects of media vs. medium, practical in vivo issues, usability, and economic feasibility.31 In addition, three general categories of technical, practical, and user-based challenges for implementing a VR-based telerehabilitation program have been discussed.32 Two main limitations of using VR in assessing balance and rehabilitation were the delay between the input and output devices, as the latency of the system as well as perceived distance in the virtual environment compared to the real situation is underestimated in VR.2 Moreover, common criticisms regarding VR for neuroscience research include cost, the requirement for specialist technical skills, the bulkiness of equipment, and cybersickness.10

In this study, we have reviewed the literature to organize the challenges into categories and sub-categories.

We suggest the following recommendations to overcome the general and specific challenges identified:

• For reduced face-to-face communications: To decrease the disadvantages, a patient at the beginning of the treatment process should first have a physical presence in the healthcare organization to visit his/her physician. If the patient needs to use VR out of the healthcare organization, it is essential to perform patient training and handle the treatment process under the professional’s supervision. Finally, to ensure the effectiveness of the programs, evaluation studies should be made.
• For education: It is necessary to educate and train clinical experts and patients in the proper, professional, and ethical use of VR as a useful tool for medical education and treatment. Choosing the type and method of training for the necessary skills to users should be done through considering the age, the level of literacy, and awareness of them. It is obvious that spending enough time and using skilled personnel to educate healthcare providers and patients have an important impact on the more effective use of VR.
• For cost: The need to design and implement therapeutic and education programs by a team of experts should be defined, prioritized, and guaranteed by the financial support of the projects to be more successful. Since producing content for VR-based programs is expensive, all stages of design and development must be evaluated because making changes to the content of the final versions is far more difficult and costly. For this purpose, the programs are first developed as prototypes, and the final version will be presented after the implementation of the pilot studies and content verification.
• For users’ attitudes: The acceptance of VR and modern technologies by patients and therapists as well as students for educational use should be considered an essential principle.
• For designing: The purpose-based and standard design of VR-based educational and therapeutic programs requires collaboration between medical and IT experts as well as end-user by their feedbacks and comments to provide the effective content and successful implementation of the programs. Using a UCD approach to design usable programs is recommended. In programs with a focus on education, students need to be aware of the limitations of interactive electronic content, such as failing to provide some details in designing programs.
• For safety considerations: Safety parameters and guidelines created are used to ensure the safety of applications. In addition, therapeutic and education programs under the supervision of specialists should be done.
• For side effects of vr: Managing and monitoring the health of users of VR programs is necessary. Moreover, diagnostic and preventive plans for the change in individuals’ behavior
  after applying VR-based games and applications should be considered. It should also be emphasized that these tools are used as complementary to the usual methods, and not a substitute for it.

• For evaluation and validation of vr applications: Science-based standards development with the collaboration of specialized teams is essential for evaluating VR systems. The continuity of evaluation, as well as the use of a combination of quantitative and qualitative methods for evaluations, especially usability evaluation, should also be considered outside the lab and in the clinical setting. Also, the responsibilities of program producers and project team should be defined in the form of guidelines and rules to specify who will be responsible in the case of a problem occurring for the users by the use of these modern tools. Although some successes and positive outcomes have been claimed for the use of VR, further studies and standard large-scale randomized clinical trials are needed. VR applications should not be used before validating programs and providing the operating instructions.

We only included papers published in English and Persian languages through searches in three electronic databases. Papers in other languages and databases, as well as grey literature, may hold valuable and reportable new information on this topic. This paper is not a systematic review. Accordingly, we did not follow the PRISMA guidelines for search. However, we have described the method and our search query. Although we do not claim that this study is comprehensive, the stated cases and our findings can help create insights to reduce possible obstacles in the projects and prevent some preventable problems. It also can even be useful as a directive for future research.

Conclusion

In Iran, VR approaches have been less applied in medicine, especially in the field of education and treatment. Lack of clinicians’ knowledge and awareness about this technology is one of the contributing factors. Identifying technical and non-technical limitations have a pivotal role in the successful and proper use of these systems. Lack of suitable standards, insufficient infrastructure, difficulties in content providing, organizational culture, and management support are some of these limitations. In general, planning for the study of experiences of the leading countries in the use of these technologies, developing and updating related laws, guidelines, standards, and utilizing appropriate models in the design and implementation of VR-based applications should help in using VR in medical education and treatment. Also, there is a need to determine appropriate policies that will play an important role in the successful implementation of these technologies at the national level.

Disclosure

The authors declared no conflicts of interest.

references

1. 1. Bun P, Gorski F, Grajewski D, Wichniarek R, Zawadzki P. Low–cost devices used in virtual reality exposure therapy. Procedia Comput Sci 2017;104:445-451.
2. 2. Morel M, Bideau B, Lardy J, Kulpa R. Advantages and limitations of virtual reality for balance assessment and rehabilitation. Neurophysiol Clin 2015 Nov;45(4-5):315-326.
3. 3. Nichols S, Patel H. Health and safety implications of virtual reality: a review of empirical evidence. Appl Ergon 2002 May;33(3):251-271.
4. 4. Mujber TS, Szecsi T, Hashmi MS. Virtual reality applications in manufacturing process simulation. J Mater Process Technol 2004;155:1834-1838.
5. 5. Guttentag DA. Virtual reality: applications and implications for tourism. Tour Manage 2010;31(5):637-651.
6. 6. Mantovani F, Castelnuovo G, Gaggioli A, Riva G. Virtual reality training for health-care professionals. Cyberpsychol Behav 2003 Aug;6(4):389-395.
7. 7. Sherman WR, Craig AB. Understanding virtual reality: interface, application, and design. San Francisco, CA: Morgan Kaufmann; 2003. p. 6.
8. 8. What is virtual reality? [cited 2017 March 31]. Available from: https://www.vrs.org.uk/virtual-reality/what-is-virtual-reality.html.
9. 9. Huang H-M, Rauch U, Liaw S-S. Investigating learners’ attitudes toward virtual reality learning environments: based on a constructivist approach. Comput Educ 2010;55(3):1171-1182.
10. 10. Bohil CJ, Alicea B, Biocca FA. Virtual reality in neuroscience research and therapy. Nat Rev Neurosci 2011 Nov;12(12):752-762.
11. 11. Mohajeri MR, Mohajeri AR. [Simulation and virtual reality; a new way to improve the quality of medical education]. Horizons of Medical Education Developement 2010;4(1):69-74.
12. 12. Parsons TD, Rizzo AA. Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis. J Behav Ther Exp Psychiatry 2008 Sep;39(3):250-261.
13. 13. Gachet Páez D, Aparicio F, de Buenaga M, Padrón V. Personalized health care system with virtual reality rehabilitation and appropriate information for seniors. Sensors (Basel) 2012;12(5):5502-5516.
14. 14. Pfandler M, Lazarovici M, Stefan P, Wucherer P, Weigl M. Virtual reality-based simulators for spine surgery: a systematic review. Spine J 2017 Sep;17(9):1352-1363.
15. 15. Schultheis MT, Rizzo AA. The application of virtual reality technology in rehabilitation. Rehabil Psychol 2001;46(3):296-311.
16. 16. Srivastava K, Das RC, Chaudhury S. Virtual reality applications in mental health: challenges and perspectives. Ind Psychiatry J 2014 Jul-Dec;23(2):83-85.
17. 17. Stull AT, Hegarty M, Mayer RE. Getting a handle on learning anatomy with interactive three-dimensional graphics. J Educ Psychol 2009;101(4):803.
18. 18. Lindner P, Miloff A, Hamilton W, Reuterskiöld L, Andersson G, Powers MB, et al. Creating state of the art, next-generation virtual reality exposure therapies for anxiety disorders using consumer hardware platforms: design considerations and future directions. Cogn Behav Ther 2017 Sep;46(5):404-420.
19. 19. Lloréns R, Noé E, Colomer C, Alcañiz M. Effectiveness, usability, and cost-benefit of a virtual reality-based telerehabilitation program for balance recovery after stroke: a randomized controlled trial. Arch Phys Med Rehabil 2015 Mar;96(3):418-425.e2.
20. 20. Cho S, Ku J, Cho YK, Kim IY, Kang YJ, Jang DP, et al. Development of virtual reality proprioceptive rehabilitation system for stroke patients. Comput Methods Programs Biomed 2014;113(1):258-265.
21. 21. Kim JH, Jang SH, Kim CS, Jung JH, You JH. Use of virtual reality to enhance balance and ambulation in chronic stroke: a double-blind, randomized controlled study. Am J Phys Med Rehabil 2009 Sep;88(9):693-701.
22. 22. Turolla A, Dam M, Ventura L, Tonin P, Agostini M, Zucconi C, et al. Virtual reality for the rehabilitation of the upper limb motor function after stroke: a prospective controlled trial. J Neuroeng Rehabil 2013 Aug;10:85.
23. 23. Salem Y, Gropack SJ, Coffin D, Godwin EM. Effectiveness of a low-cost virtual reality system for children with developmental delay: a preliminary randomised single-blind controlled trial. Physiotherapy 2012 Sep;98(3):189-195.
24. 24. Gandolfi M, Geroin C, Dimitrova E, Boldrini P, Waldner A, Bonadiman S, et al. Virtual reality telerehabilitation for postural instability in Parkinson’s disease: a multicenter, single-blind, randomized, controlled trial. Biomed Res Int 2017;2017:7962826.
25. 25. Malloy KM, Milling LS. The effectiveness of virtual reality distraction for pain reduction: a systematic review. Clin Psychol Rev 2010 Dec;30(8):1011-1018.
26. 26. Nunes FL, Costa RM. The virtual reality challenges in the health care area: a panoramic view. In: Proceedings of the 2008 ACM symposium on Applied computing; 2008; New York: ACM; 2008. p. 1312-1316.
27. 27. Rogers L. Developing simulations in multi user virtual environments to enhance healthcare education. Br J Educ Technol 2011;42(4):608-615.
28. 28. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002 Oct;236(4):458-463, discussion 463-464.
29. 29. Ammenwerth E, Shaw NT. Bad health informatics can kill–is evaluation the answer? Methods Inf Med 2005;44(1):1-3.
30. 30. Horsky J, Zhang J, Patel VL. To err is not entirely human: complex technology and user cognition. J Biomed Inform 2005 Aug;38(4):264-266.
31. 31. Garrett B, Taverner T, Gromala D, Tao G, Cordingley E, Sun C. Virtual reality clinical research: promises and challenges. JMIR Serious Games 2018 Oct;6(4):e10839.
32. 32. Rizzo AA, Strickland D, Bouchard S. The challenge of using virtual reality in telerehabilitation. Telemed J E Health 2004;10(2):184-195.
33. 33. Teixeira L, Ferreira C, Santos BS. User-centered requirements engineering in health information systems: a study in the hemophilia field. Comput Methods Programs Biomed 2012 Jun;106(3):160-174.
34. 34. Baniasadi T, Niakan Kalhori SR, Ayyoubzadeh SM, Zakerabasali S, Pourmohamadkhan M. Study of challenges to utilise mobile-based health care monitoring systems: a descriptive literature review. J Telemed Telecare 2018 Dec;24(10):661-668.
35. 35. Lluch M. Healthcare professionals’ organisational barriers to health information technologies-a literature review. Int J Med Inform 2011 Dec;80(12):849-862.
36. 36. Rizzo AA, Schultheis MT, Rothbaum B. Ethical issues for the use of virtual reality in the psychological sciences. In: Bush S, Drexler M, editors. Ethical issues in clinical neuropsychology: Lisse, NL: Swets & Zeitlinger; 2002. p. 243-280.
37. 37. Stanberry B. Legal ethical and risk issues in telemedicine. Comput Methods Programs Biomed 2001 Mar;64(3):225-233.
38. 38. Botella Arbona C, García Palacios A, Baños Rivera RM, Quero Castellano S. Cybertherapy: Advantages, limitations, and ethical issues. PsychNology J 2009;7(1):77-100.
39. 39. Lockwood D. Evaluation of virtual reality in Africa: an educational perspective. Paris: UNESCO; 2004.
40. 40. Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review. Int J Med Inform 2013 May;82(5):e73-e86.
41. 41. Mohammadzadeh N, Safdari R. Patient monitoring in mobile health: opportunities and challenges. Med Arch 2014;68(1):57-60.
42. 42. Venkatesh V, Thong JY, Xu X. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. Manage Inf Syst Q 2012;36(1):157-178.
43. 43. Al-Mujaini A, Al-Farsi Y, Al-Maniri A, Ganesh A. Satisfaction and perceived quality of an electronic medical record system in a tertiary hospital in Oman. Oman Med J 2011 Sep;26(5):324-328.
44. 44. Von Jan U, Noll C, Behrends M, Albrecht U-V. MARble–augmented reality in medical education. Biomed Tech (Berl) 2012;57(SI-1 Track-A):67-70.
45. 45. Sutcliffe A, Gault B. Heuristic evaluation of virtual reality applications. Interact Comput 2004;16(4):831-849.
46. 46. Sutcliffe AG, Kaur KD. Evaluating the usability of virtual reality user interfaces. Behav Inf Technol 2000;19(6):415-426.
47. 47. Johnson CM, Johnson T, Zhang J. Increasing productivity and reducing errors through usability analysis: a case study and recommendations. Proc AMIA Symp 2000:394-398.
48. 48. Zhang J, Patel VL, Johnson KA, Smith JW. Designing human-centered distributed information systems. IEEE Intell Syst 2002;17(5):42-47.
49. 49. Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform 2005 Feb;38(1):75-87.
50. 50. Koch S. Designing clinically useful systems: examples from medicine and dentistry. Adv Dent Res 2003 Dec;17(1):65-68.
51. 51. Kujala S. User involvement: a review of the benefits and challenges. Behav Inf Technol 2003;22(1):1-16.
52. 52. Kushniruk A, Nøhr C. Participatory design, user involvement and health IT evaluation. Stud Health Technol Inform 2016;222:139-151.
53. 53. Beuscart-Zéphir MC, Brender J, Beuscart R, Ménager-Depriester I. Cognitive evaluation: how to assess the usability of information technology in healthcare. Comput Methods Programs Biomed 1997 Sep;54(1-2):19-28.
54. 54. Carroll C, Marsden P, Soden P, Naylor E, New J, Dornan T. Involving users in the design and usability evaluation of a clinical decision support system. Comput Methods Programs Biomed 2002 Aug;69(2):123-135.
55. 55. Vimarlund V, Timpka T. Design participation as an insurance: risk-management and end-user participation in the development of information systems in healthcare organizations. Methods Inf Med 2002;41(1):76-80.
56. 56. Rinkus S, Walji M, Johnson-Throop KA, Malin JT, Turley JP, Smith JW, et al. Human-centered design of a distributed knowledge management system. J Biomed Inform 2005 Feb;38(1):4-17.
57. 57. Zhao Q. 10 scientific problems in virtual reality. Commun ACM 2011;54(2):116-118.
58. 58. Duan Z, Yuan Z, Liao X, Si W, Zhao J. 3D tracking and positioning of surgical instruments in virtual surgery simulation. J Multimed 2011;6(6).
59. 59. Welch G, Foxlin E. Motion tracking: no silver bullet, but a respectable arsenal. IEEE Comput Graph Appl 2002;22(6):24-38.
60. 60. Grajewski D, Górski F, Zawadzki P, Hamrol A. Application of virtual reality techniques in design of ergonomic manufacturing workplaces. Procedia Comput Sci 2013;25:289-301.
61. 61. Robillard G, Bouchard S, Fournier T, Renaud P. Anxiety and presence during VR immersion: a comparative study of the reactions of phobic and non-phobic participants in therapeutic virtual environments derived from computer games. Cyberpsychol Behav 2003 Oct;6(5):467-476.
62. 62. Herbelin B, Vexo F, Thalmann D. Sense of presence in virtual reality exposures therapy. Proceedings of the 1st International Workshop on Virtual Reality Rehabilitation Lausanne, Switzerland: Citeseer; 2002.
63. 63. Carnegie K, Rhee T. Reducing visual discomfort with HMDs using dynamic depth of field. IEEE Comput Graph Appl 2015 Sep-Oct;35(5):34-41.
64. 64. Serge SR, Moss JD. Simulator sickness and the oculus rift: a first look. Proceedings of the Human Factors and Ergonomics Society 59th Annual Meeting, 2015, Santa Monica 2015;59(1):761-765.DOI: 10.1177/1541931215591236.
65. 65. Garrett B, Taverner T, McDade P. Virtual reality as an adjunct home therapy in chronic pain management: an exploratory study. JMIR Med Inform 2017 May;5(2):e11.
66. 66. Chiu S-I, Lee J-Z, Huang D-H. Video game addiction in children and teenagers in Taiwan. Cyberpsychol Behav 2004 Oct;7(5):571-581.
67. 67. Tejeiro Salguero RA, Morán RM. Measuring problem video game playing in adolescents. Addiction 2002 Dec;97(12):1601-1606.
68. 68. Young K. Understanding online gaming addiction and treatment issues for adolescents. Am J Fam Ther 2009;37(5):355-372.
69. 69. Rizzo AA, Kim GJ. A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence (Camb Mass) 2005;14(2):119-146.
70. 70. Khajouei R, Azizi AA, Atashi A. Usability evaluation of an emergency information system: a heuristic evaluation. Journal of Health Administration 2013;16(52):61-72.
71. 71. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform 2005 Aug;74(7-8):519-526.
72. 72. Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: guidance on usability. 1998 [cited 2018 February 6]. Available from: https://www.iso.org/standard/16883.html.
73. 73. Virvou M, Katsionis G. On the usability and likeability of virtual reality games for education: the case of VR-ENGAGE. Comput Educ 2008;50(1):154-178.
74. 74. Ni LT, Fehlings D, Biddiss E. Design and evaluation of virtual reality-based therapy games with dual focus on therapeutic relevance and user experience for children with cerebral palsy. Games Health J 2014 Jun;3(3):162-171.