Health worker experiences on the integration of digital health tools for HPV vaccination and cervical cancer services in Rwanda

Abstract

Background

Cervical cancer poses a significant global public health concern, especially in low- and middle-income countries like Rwanda, where access to preventive measures and screening is limited. The World Health Organization (WHO) urges nations to intensify efforts in human papillomavirus (HPV) vaccination, screening, and cervical cancer treatment. However, challenges in implementation persist. Digital health interventions have gained attention as potential solutions to enhance the effectiveness and accessibility of these interventions. This study assesses users’ experiences, including acceptability, satisfaction, and integration of digital health interventions for HPV vaccination and cervical cancer screening in Rwanda.

Methodology

Using a qualitative approach, the study engaged 15 participants using purposeful sampling to ensure data saturation and maximum variation. Semi-structured, face-to-face interviews were conducted with key informants in selected Rwandan healthcare facilities that currently use digital health interventions for HPV vaccinations (e-tracker), and cervical cancer screening and treatment (mUzima app). Interviews with key informants were audio-recorded, transcribed, and manually coded for thematic analysis to extract common themes and patterns. Anonymous quotes were used to illustrate these themes.

Results

User experiences with digital health interventions, particularly regarding usability, acceptability, and satisfaction, were largely positive. Nevertheless, transitioning from paper-based to digital systems has presented challenges, including limited computer literacy among users, initial resistance to change, and a shortage of necessary equipment. Factors such as leadership commitment, technical support, and supervision have been critical in the successful implementation of these digital interventions.

Conclusion

The study offers valuable insights into the advantages, obstacles, and methods to improve adoption and effectiveness of digital health interventions in cervical cancer prevention, aiming to reduce the disease burden in Rwanda. The findings also provide potential global insights for similar initiatives in cervical cancer prevention, suggesting broader applicability and significance of this research in other contexts.

Background

Cervical cancer remains a significant global public health concern, particularly in low- and middle-income countries (LMICs) such as Rwanda, where access to preventive measures and screening services is often limited [1,2,3]. Cervical cancer ranks as the fourth most common cancer among women worldwide, with approximately 604,000 new cases and 34,200 deaths in 2020, 90 percent of which occurred in LMICs [1, 4]. Nearly all cervical cancer cases are attributed to HPV [1].

Comprehensive cervical cancer control includes primary prevention (HPV vaccination), secondary prevention (screening and treatment of precancerous lesions), tertiary prevention (diagnosis and treatment of invasive cervical cancer), and palliative care [1]. In 2018, the WHO issued a call to action for countries to collaborate toward eradicating cervical cancer through improved efforts in HPV vaccination, screening, and treatment. These endeavors aim to achieve the 90–70–90 targets, which include attaining 90 percent HPV vaccination coverage for girls by age 15, screening 70 percent of women by ages 35 and 45, and treating 90 percent of pre-cancer cases and managing 90 percent of invasive cancer cases by 2030, with the overarching goal of achieving elimination within the next century [5]. However, implementing interventions to accelerate HPV vaccination and cervical cancer screening include numerous challenges, including limited awareness, accessibility, and adherence to vaccination schedules and screening guidelines [6,7,8].

Digital health interventions have increasingly been recognized as potential solutions to these challenges, enhancing the effectiveness of delivery services and accessibility of HPV vaccination and cervical cancer screening [9, 10]. Digital health interventions encompass various technologies, such as mobile apps, online platforms, and electronic tracking systems, which support information sharing, appointment reminders, education materials, and personalized reminders for vaccination and screening [9,10,11].

In healthcare settings, user acceptability and satisfaction are key factors in the successful implementation and ongoing engagement with these technologies [12,13,14,15]. User-centered design and feedback are essential for improving digital health interventions acceptability and uptake [16].

Successful integration and adaptation of digital health interventions within existing healthcare frameworks are critical for their scalability and sustainability [12]. Furthermore, there is a need for strong partnerships between digital health intervention developers, healthcare providers, and policymakers to ensure seamless integration and sustainability [17].

Factors facilitating the adoption and sustained use of digital health interventions include political commitment, positive user experiences, perceived usefulness, supportive healthcare provider recommendations, interactivity, user-centered design, and integration into existing systems [12, 16, 18]. On the other hand, barriers such as limited digital literacy, data privacy and security concerns, and resistance to change have been identified [12, 18]. Understanding these country-specific barriers and facilitators is vital for successful digital health intervention implementation.

In 2019, Rwanda introduced the electronic vaccination tracker (e-tracker) built in the existing digital health platform to capture and manage all childhood vaccinations [19,20,21]. It was only in 2022 that the HPV vaccine module was added to enable digital data capture for HPV vaccination services. Since 2020, 28 district hospitals and two referral hospitals in Rwanda providing cervical cancer services have been using the mUzima application for data management, specifically for cervical cancer screening purposes. This mobile application, with both online and offline functionalities syncs with the Open Medical Record System (OpenMRS), a web-based application, enabling data access across different facilities [22].

To achieve the 90–70–90 vaccination, screening, and treatment targets outlined in the WHO cervical cancer elimination strategy, Rwanda sought to strengthen these data systems for effective progress monitoring. Over the last five years the government of Rwanda has been championing digital transformation through the ICT sector strategic plan 2018–2024. With support from partners, the Rwanda Ministry of Health has been able to enhance and introduce new digital health interventions that focus on advancing cervical cancer elimination in the country. These digital health interventions include introduction of the electronic HPV registry system in 2022, enhancement of mUzima system to exchange data with the cancer referral hospital and enhance the national cancer registry to automate data exchange with national Cancer reference hospital (Butaro).

This study aimed to evaluate the users’ experience including acceptability, satisfaction, and integration of digital health interventions for HPV vaccination and cervical cancer screening in Rwanda. The study’s objectives were: 1) To explore and understand the experiences of end-users with digital health interventions for HPV vaccination and cervical cancer services in Rwanda, 2) To assess end-user acceptability and satisfaction with these digital health interventions, 3) To evaluate the practicability, adaptation, and integration of HPV vaccination and cervical cancer digital health interventions within the existing routine workflows of Rwanda’s healthcare system, and 4) To further explore the barriers and facilitators affecting the adoption and utilization (implementation) of these digital health interventions.

Methodology

Research settings

This research was conducted in Rwanda, a landlocked nation situated in the heart of Africa. It is categorized as a low-income country with a Gross Domestic Product (GDP) of 13.3 billion Rwandan Francs (RWF) reported in 2022. The Rwanda population is estimated to be over 13.7 million with a life expectancy at birth of around 69 years as of 2022 [23]. Administratively, Rwanda is divided into 30 districts, which are further organized into four provinces (South, North, East, and West) in addition to the capital city, Kigali. These districts are subdivided into sectors, which in turn are divided into cells, and the smallest administrative unit is the umudugudu (village).

The Rwandan healthcare system functions through both public and private sectors. The public sector is structured across three levels: the central level, encompassing the Ministry of Health and its affiliated institutions as well as referral hospitals; the intermediate level, consisting of provincial and district hospitals; and the peripheral level, which includes health centers, health posts, and community health workers[24].

In Rwanda, cervical cancer is the most frequently diagnosed cancer among women, with an annual incidence rate of 42 cases per 100,000 women [25, 26]. The country has led the way with a nationwide program for cervical cancer prevention, particularly through HPV vaccination which was introduced in 2011 for girls around 12 years old [26]. Additionally, screening and vaccination are provided at all primary levels which include health centers, health posts, and community services such as school-based vaccination programs [3, 27]. Biopsies and treatment for precancerous lesions are available at district hospitals. Invasive advanced cancers are referred to tertiary level facilities for treatment where biopsies and pap smears are also conducted [3, 28].

In 2019, Rwanda launched the electronic vaccination tracker (e-tracker) to oversee childhood immunizations, later expanding it in 2022 to include HPV vaccination services. Currently, healthcare providers and data managers at healthcare facilities use this system to manage HPV vaccinations. Since 2020, the mUzima app has been implemented in 28 district hospitals and two referral hospitals for cervical cancer screening and treatment, with healthcare providers and data managers integrating it into the OpenMRS platform. Both digital health tools are monitored and managed by the Rwanda Biomedical Center (RBC).

Study design and study period

This qualitative study used semi-structured interviews conducted face-to-face with key informants. The interviews, steered by an interview guide, aimed to delve into the acceptability, satisfaction, integration, and user experiences of digital health interventions deployed in HPV and cervical cancer services. The qualitative data collection was conducted from October 3, 2023, to October 5, 2023.

Study settings and participants

The research was carried out in selected healthcare facilities in Rwanda, actively involved in HPV vaccination and cervical cancer services, including screening and treatment. Key informant interviews were conducted among healthcare providers directly engaged in the administration and management of HPV vaccination programs, screening, and management of cervical cancer, as well as data managers of these services. In addition, program managers and monitoring and evaluation(M&E) officers at central levels were also interviewed.

Sampling

We used a purposeful sampling strategy to select approximately 15 participants for the semi-structured interviews. Specifically, our target population consisted of healthcare providers and program managers directly or indirectly involved in HPV and cervical cancer programs. To ensure a varied and heterogeneous sample, we referred to a sampling matrix outlined in Table 1, designed based on the principle of maximum variation. This approach allowed us to capture diverse participant characteristics and experiences. We included healthcare providers from different cadres and with varying levels of experience working within the healthcare system, as well as the program managers from the central level (Rwanda Biomedical Center) involved in HPV and cervical cancer programs. This comprehensive sampling method aimed to achieve maximum variation and enhance the depth of our study.

Three health centers (Kicukiro, Kibuye, and Hanika) were chosen, and five key informants from these health centers involved in HPV vaccination, cancer screening, and data management using the e-tracker were interviewed. A district hospital was also chosen, where two key informants (one nurse and one data manager) were interviewed. Additionally, at Butaro Hospital, four key informants (a nurse, data officer, data manager, and doctor) utilizing mUzima and Open-MRS Oncology in the provision of cervical cancer services were interviewed. The selection of these health centers and hospitals was deliberate, aiming for diverse representation across a spectrum extending from Kigali capital city to peripheral health facilities. These institutions were chosen based on their substantial involvement in both vaccination efforts and the provision of cervical cancer services, thus reflecting a comprehensive cross-section of healthcare institutions.

At the central level, four key informants involved in HPV vaccinations and cervical cancer programs, data management, and M&E (National Cancer Registry, HPV e-tracker, mUzima/OpenMRS, Health Information and Management System: HIMS) were interviewed—two from the immunization program and two from the noncommunicable diseases/cancer division.

Data collection

The principal investigator (PI) provided one hour of training to review the interview guide and discuss qualitative interview techniques, including reviewing additional prompts. Four interviewers received training in interviewing techniques and methods to the data collection in advance. These interviews were carried out in quiet, private settings and were conducted in either Kinyarwanda or English, depending on the language preference of the interviewee. All interviews were audio-recorded for accuracy and thoroughness.

The interview guide was developed through a collaborative effort with significant input from the PI who shaped the initial framework based on his expertise in the field. Additionally, feedback and suggestions from other research team members were sought and incorporated to refine the guide before the pilot phase. Piloting of the interview guide involved conducting a dry run with a small group of three individuals, selected separately from those who participated in this study. This process aimed to detect any limitations in question interpretation and interview skills, thereby determining the need for changes in the questions such as rewording, rephrasing, or use of probes to enhance clarity and bring out rich detail in the responses. Interviewees were asked about their experiences, challenges, and suggestions for improving the practicability and integration of the digital health interventions for HPV and cervical cancer. The interview guides can be found in the supplementary materials. Each participant was compensated with the equivalent of $10 in Rwandan francs as an incentive upon completion of the interviews. Face-to-face semi-structured interviews, each lasting approximately 30–45 min, were conducted with eligible healthcare providers using an interview guide. Data collection continued until thematic saturation was reached after 13 interviews, as no new themes or concepts emerged during the 14th and 15th interviews. This study was approved by the Rwanda National Ethics Committee (RNEC) under Approval Notice No. 156/RNEC/2023.

Data management and data analysis

Audio recordings from the interviews were captured using a voice recorder and then securely transferred to a password-protected digital device, accessible only to the PI, the transcribers/translators, and other research team members. Audio recordings were transcribed verbatim and translated accurately into English as needed. For interviews originally conducted in Kinyarwanda, back-translation was performed to ensure that the translations accurately conveyed the intended meanings and explanations.

Interview transcripts were managed and organized into Word documents and Excel spreadsheets for analysis. Data analysis consisted of familiarization with data through careful reading and re-reading of transcripts by the PI and two qualitative researchers to identify preliminary codes.

The qualitative researchers independently coded five transcripts concurrently and subsequently convened with the PI to review and refine the codes, leading to the development of a codebook. The codebook underwent a “member checking” process with the research team members who were involved in the qualitative data collection and the PI, ensuring an appropriate interpretation.

Once the final codebook was validated by the PI, the two qualitative researchers proceeded to code the remaining transcripts using the established codebook. Key concepts derived from the codes were grouped together to form categories and subcategories and linkages made by identifying themes and sub-themes for charting, mapping, and linking to central themes. Anonymous quotations by participant type (i.e., nurse) were used to exemplify the themes.

The quality of qualitative data was maintained through adherence to the principles of credibility, transferability, and dependability. The credibility of the data was enhanced by re-reading the transcripts, re-listening to the audio, and confirming data consistency to ensure that there were no missed perspectives. Transferability was bolstered by collecting detailed demographic information and providing rich descriptions of the study setting, participants, and observed practices, thereby enabling potential application to different studies or settings. A confirmatory inquiry to respective research assistants involved in the data collection further strengthened the dependability of the data.

Results

Demographics of key informant interview participants

Fifteen semi-structured interviews were conducted with healthcare providers, data managers, administrators, and program managers involved in HPV vaccination programs, screening, and management of cervical cancer. Most participants were nurses and data managers directly engaged in providing these services, with most working at urban healthcare facilities. The participants had an average professional experience of 11 years. (See Table 2a).

Table 1 Characteristics of study participants

Table 2 Summary of key themes and subthemes identified in the qualitative analysis

Findings from key informant interviews

The analysis of semi-structured interviews with key informants revealed four overarching themes, each shedding light on crucial aspects of digital health interventions in the context of an HPV vaccination program, and cervical cancer screening and management. From the findings, rich descriptions and participant quotations were pulled to illustrate these themes, providing further depth and context. The four identified themes were:

• Theme 1: End-user experience, acceptability, and satisfaction with digital health interventions.

• Theme 2: Systemic and operational challenges in the adaptation and integration of digital health interventions within existing healthcare systems.

• Theme 3: The need for investment in personnel and development of data management capacities for digital health interventions.

• Theme 4: The critical role of leadership commitment, technical support, and supervision in facilitating digital health intervention success.

Theme 1: End-user experience, acceptability, and satisfaction with digital health interventions

Although mixed opinions were expressed, a recurring theme from the semi-structured interviews was the overall positive end-user experience with digital health interventions. Most participants consistently reported positive experiences, high acceptability, and overall satisfaction. The convenience of these technologies was highlighted, with users expressing confidence and ease in navigating the digital platforms. Key features of the digital health interventions were:

• a Ease of use

  The users expressed their appreciation for the general user-friendly nature of the digital systems, particularly after receiving training, which made the use even easier. The initial cohorts formally trained, smoothly navigated the digital systems and supported their co-workers in learning how to use them.

  “The interface is user-friendly; once you are trained, you should have no difficulty using it.”– KII (1), Nurse

  “OpenMRS is user-friendly and generally easy to use; you just limit it to the functionality that you need for your job.” – KII (6), Medical doctor

  The ease of use was expressed by even those with low digital literacy who were with time able to overcome this challenge and are now competent in using the system.

  “When e-tracker (e HPV tracker) was introduced, it was difficult for me initially due to my limited computer literacy. However, with time, I got used to it. As of today, I no longer face any difficulties, except when the internet connection is slow or cut off. During such times, we use the register and enter the data into the system once the internet is back. However, this doesn't mean we maintain parallel systems; we exclusively use e-trackers” – KII (4)_ Nurse

  “Because nurses are the main end users, initially they lacked sufficient computer literacy. However, with time, they became familiar with the system and realized that it helps them better than using registers.” KII (9)_Program manager

  Additionally, they valued the offline capability of these systems, as it streamlines their work, allowing for synchronization even when operating outside health facilities with unreliable internet connections. This sentiment was shared by both end-users and program managers at the central level.

  “A positive example of feedback is that users are very pleased and satisfied with the mUzima system. They appreciate its ability to work offline, facilitating operations in areas where internet connectivity is inconsistent. Additionally, they value its synchronization with OpenMRS, leading to a high level of satisfaction among our users.”– KII (15), Program manager

• b Improvement over paper registries and enhanced data quality and security

  Participants emphasized the efficiency brought about by digitalization in simplifying scheduling appointments, tracking follow-ups, and ultimately enhancing coverage.

  “Yes, the e-tracker has been incredibly helpful. The ability to know who is scheduled to come each day allows us to proactively reach out to those who miss their appointments and ensure they receive the vaccine. This is a significant improvement from the time when we used registers. It was challenging to go through multiple registers to identify those who didn't attend. Thanks to e-tracker, our efficiency has increased, leading to a higher number of vaccinations and improved coverage.” – KII (1), Nurse

  “I am happy and satisfied, rating my experience at 10/10. Previously, I was using 5 registers, but now, with just one laptop, I can manage everything.”– KII (3), Nurse

  Key informants recognized the implication of digital health interventions in enhancing data storage and management. They noted that previously, when they relied on paper registers, retrieving prior information was not guaranteed and required searching through numerous paper registers, a process that was time-consuming. However, with the implementation of digital systems, accessing patient’s information became remarkably simpler, requiring only the patient identifier. This streamlined approach enhances data quality by minimizing the risk of information loss through loss or damage of paper records and ensures data security.

  “The use of the mUzima and OpenMRS system has significantly eased our processes and increased data security compared to the time when we relied on papers and registers.” – KII (7), Data manager

  “Regarding the safety and security of data, we can say yes. You cannot use the system without the proper credentials. As long as the credentials are kept confidential and not shared, the security of the data is ensured” KII (8), Data manager

  “Paper-based systems have too many errors, as people can add information at any time. In contrast, with e-tracker, if someone makes changes, you can track and even know when the changes were made. Data quality has significantly improved, and discrepancies have decreased” - KII (2), Data manager

  “Using registers was risky; the papers could be torn, lost, or damaged. But now, with e-tracker, you have secure data that won't be lost” – KII (2), Data manager

• c Time saving and increased access to data

  Thirteen of the 15 participants indicated that digital health interventions were beneficial for both providers and patients. These interventions have significantly improved efficiency, reducing the time spent serving individual clients and thereby increasing the number of clients attended per day. Additionally, they have facilitated rapid and timely reporting. These efficiencies have also led to reduced waiting times for patients at health facilities, a change that has been well-received.

  “During that time, parents used to complain because they had to spend more time at the health center for vaccinations, waiting for providers to fill in both the registers and the e-tracker. However, today they are pleased with the system. It is faster, easier, and the time spent in the health center for vaccinations has decreased. It's a source of satisfaction for both parents and providers.” – KII (1), Nurse

  “Initially, when e-tracker was launched, we were not happy because we thought it would double our workload. We had to use both the registers and the e-tracker. However, now we only use e-tracker, and this has significantly improved our efficiency. The sessions, which used to last six hours, now only take about four hours. Both parents and providers are satisfied with this process. – KII (3), Nurse

  “E-tracker significantly enhances data management and ensures the quality of stored information. It enables rapid tracking of a child's vaccination history. In the past, using registers meant searching through numerous records, often taking hours, especially if the parent couldn't recall the vaccination year. However, with the unique identifier code, all the necessary information is readily available.” – KII (2), Data manager

  “E-tracker is very effective; it helps me produce reports on time.” – KII (2), Data manager

  The use of a digital system, in addition to enhancing data storage, quality and security meaningfully improved the availability of data for various purposes. First, patient information necessary for proper follow-up became readily accessible through systems that allowed for synchronization and interoperability. Second, researchers benefited from the availability of high-quality data due to the utilization of these digital systems.

  “mUzima is a great system because it allows synchronization. The data entered in mUzima is transferred to OpenMRS. All screenings are registered in mUzima and synchronized, then transferred to OpenMRS oncology. It helps us a lot as data managers to do triangulation, as we can access all information using the same patient codes. We no longer need transfer sheets for patients registered in mUzima; the data entered at the health center can be accessed at the district hospital level.” -– KII (8), Data manager

  “I would also like to mention the significant impact on research. These systems have facilitated access to valuable data for researchers, enabling them to conduct studies effectively. Additionally, the systems assist us in planning by providing insights into patient demographics and trends. This foresight helps us project the resources, materials, and equipment we need, allowing us to make timely acquisitions.” – KII (14), Program manager

• d Diverse perspectives on the digital health interventions

  Despite the generally positive experiences reported, a minority of interviewees expressed mixed reviews about digital health interventions. These digital interventions were generally well received by healthcare providers and, by extension, the clients attending the healthcare facilities where these interventions are implemented. The overall rating for the digital health interventions averaged eight on a scale of 1–10. However, mixed appraisal surfaced when providers rated different digital health interventions with preferences varying based on ease of use and functionality.

  “I would rate the open MRS system below average as I don't use it often. In contrast, mUzima's rating is above average because it's more user-friendly. However, I think that the rating depends on each providers ‘experience with the system.” – KII (10), Nurse

  Additionally, some providers exercised caution in solely attributing the increased immunization coverage to the introduction of digital health interventions, as they acknowledge the influence of other contributing factors.

  “They help in follow-up and service delivery, but I don’t think it has contributed to an increase in the number of first screenings. I believe other campaigns have contributed to this increase.”- KII (8), Data manager

  “We can't conclude that the e-tracker system itself contributed to increased coverage, but it does aid in follow-up and reduces the number of missed appointments. We can even track where a person has taken the second dose if they changed the vaccination site.”- KII (9), Program manager

  Generally, there is positive user acceptance of the different interventions across all user groups: nurses who primarily use in real time the HPV e tracker, mUzima, OpenMRS system, the data manager who sometimes are used as backstop support and super user to provide technical assistance to nurses whenever needed. The data managers are also responsible for entering data into the national cancer registry system for confirmed invasive cancer cases.

Theme 2: Systemic and operational challenges in the adaptation and integration of digital health interventions within existing healthcare systems

Key informants pointed out the systemic and operational challenges that they faced during the adaptation and integration of digital health interventions within existing healthcare systems. The transition from paper-based to digital-based systems was not entirely smooth. Some of the challenges encountered included resistance from end-users, limited computer literacy, limited digital infrastructure, lack of technical support, and poor internet connectivity.

“In the initial stages of implementation, we encountered resistance from users who were unwilling to change. However, with time, they adapted, and now they are satisfied.”– KII (12), Data scientist.

“When e-tracker was introduced, it was difficult for me initially due to my limited computer literacy. However, with time, I got used to it”– KII (4), Nurse

“In the beginning, we faced problems with equipment. As you know, at the health center, laptops or computers were limited mainly in the head of the health center's office, with the accountant, and the data manager. It was challenging for us to find a computer for data entry, especially when we went out for vaccinations in the field. Carrying a desktop in the field with electricity issues was impractical and difficult.” – KII (1), Nurse

“The challenges that we faced were financial constraints, such as funds to buy servers and equipment. Secondly, some providers had low computer literacy, which affected the quality of data at the beginning of the implementation. Thirdly, the internet was not widely available in the country. Finally, maintenance and continuous technical support were not strong at that time. Another challenge in the integration was the change in the mindset and perceptions of providers to accept the change and adopt new technology.” – KII (14), Program manager

For most of these challenges solutions were found but some challenges, like poor internet connectivity continue to exist.

“As of today, I no longer face any difficulties, except when the internet connection is slow or cut off. During such times, we use the register and enter the data into the system once the internet is back. However, this doesn't mean we maintain parallel systems; we exclusively use e-trackers” – KII (4), Nurse

Theme 3: The need for investment in personnel, training, and development of data management capacities for the digital health interventions

Informants stressed the necessity of investing in personnel and the development of robust data management capacities. They indicated that capacity for data management continues to be a challenge in the adoption of digital health interventions. This includes training and support for the process of data entry, data entry personnel, and some functionalities of the data management system in the digital interventions. There is a need to have more data entry staff, healthcare workers need training and greater attention should be paid to data entry to ensure thoroughness and completeness. Additionally, some features of the systems need to be upgraded. These features include interoperability and development of dashboards to enhance tracking.

• a Staffing

  Some healthcare providers and central-level staff mentioned the need for more dedicated personnel to use and monitor these digital health interventions. While this approach was suggested to improve efficiency, it is not common to all healthcare facilities.

  “Currently, when our nurses go for vaccinations, they are not accompanied by data entry personnel. This creates a problem as one person is responsible for administering vaccinations and then going back and forth to enter data into e-tracker…..It would be more efficient if the nurses were accompanied by a data manager. This way, one person could enter people’s details into e-tracker while the other administers the vaccines. Having both roles separated in this manner would rationalize the process and make it more effective”.– KII (4), Nurse

  “Additionally, at the central level, we require a dedicated staff member for the follow-up, maintenance, and monitoring of these systems. It is a demanding job that should be undertaken by someone exclusively dedicated to it without mixing it with other duties.”– KII (15), Program manager

• b System functionalities

  Participants made suggestions regarding the systems’ functionalities and features that need improvement, particularly focusing on interoperability and full online options.

  “In my perspective, although we are pleased that the system can work offline, there is a risk involved. If a person works offline the whole day and loses or breaks the tablet before synchronization, all data could be lost. It would be beneficial if the system could go online to ensure data entry and storage is directly sent to the central server, allowing real-time data. We also need to rigorously restrict and control duplicates as people might forget to synchronize, leading to data discrepancies.”–KII (15), Program manager

  “We also aim to develop dashboards that can be easily accessible. Another aspect is to work on interoperability between the existing systems so that patient information can be easily tracked even if she changes the health facility that uses a different system. For instance, if a patient consults and is found to have cervical cancer, the provider should be able to track her vaccination history and screening history due to the interoperability of the systems.”– KII (15), Program manager

• c Ongoing capacity building for digital health interventions

  Eight of the 15 participants requested regular training to update them on new functionalities and expand their knowledge in order to fully utilize digital health interventions.

  “Considering that we have received different training as providers, I would like to request more training on additional functionalities that we need but can't use because we do not yet master them”– KII (13), Nurse

Theme 4: Critical role of leadership commitment, technical support, and supervision in facilitating digital health intervention success

One of the key enablers of effective implementation of digital health interventions have been identified as leadership commitment, political will, responsive technical support, and robust supervision. Leadership commitment and politics were described as pivotal, providing the necessary drive and resources for digital health interventions. Establishment of digital transformation directorate at MOH ICT and digital health unit with MOH streamline digital health governance at government as whole and health sector in specific. The success of these interventions heavily relies on technical support that is attentive to the feedback of end-users and offers them prompt assistance. Equally important was the role of routine monitoring and supportive supervision, ensuring that digital health interventions are effectively integrated and utilized within the healthcare system. These facilitators operate within the context of Rwanda’s well-established and strong healthcare infrastructure, which forms the backbone supporting the successful deployment of digital health interventions.

“First, the leadership played a key role in facilitating the transition from a paper-based system to e-tracker. Once we had leadership commitment, the implementation became easy. Secondly, we conduct regular routine monitoring. We collaborate with the vaccinating health centers; they provide us with the target number of scheduled children, and we compare it with the number they have vaccinated and entered in e-tracker. This helps in verification, data completeness, and encourages continuous improvement.” – KII (9), Program manager

“One key factor that facilitated implementation was the commitment of leadership, from higher levels down to the peripheral level. The pre-existing strong health system aided integration and successful implementation. Additionally, supportive partners assisted the Ministry of Health in the implementation process. All these factors contributed to the successful implementation and sustainability” – KII (12), Data scientist

Feedback from end users serves as a valuable source of information for refining and improving the implementation of digital health interventions, demonstrating the central level’s commitment to responsive and user-centered approaches, leading to more successful implementation of digital health interventions.

“Yes, we do receive feedback from the health providers. We have a platform that allows us to meet regularly; we conduct regular quality improvement meetings with the providers. They are happy and have benefited from this system. It is easy for them to do patient follow-ups, create reports, and access data. They also report some complaints that need to be fixed. Although there are not many, we work closely with the developers to improve the functionality of these systems.” – KII (15), Program manager

Discussion

This study evaluated the experiences of end-users in utilizing digital health interventions (e-tracker and mUzima and OpenMRS) for HPV vaccination and cervical cancer services in Rwanda. It assessed the acceptability, practicability, and integration of these interventions within the existing healthcare systems, while also identifying facilitating factors and barriers to their implementation. The overall experience of end-users was generally positive. They described successfully integrating the digital health interventions into their routine work and praised the positive changes these interventions brought to their workflow and more importantly, their patients. This was despite initial use hesitancy due to concerns about increased workload. Initially, participants faced challenges with integrating the new technology, but quickly found solutions and overcame many process obstacles, although some issues, such as poor internet connectivity in certain regions, persisted. Support from the central level government, along with institutional leadership commitment and a robust healthcare system, facilitated the integration and successful implementation of these interventions.

The implementation of digital health interventions, specifically the e-tracker significantly transformed the landscape of HPV vaccination and cervical cancer services in Rwanda. Since the initiation of e-tracker in 2020 for routine childhood vaccinations and subsequent addition of HPV vaccination capabilities in 2022, the e-tracker system has played a pivotal role in data management and healthcare delivery in Rwanda [21]. All childhood immunizations are administered at the health centers, transitioning from paper-based to a completely digital system. In Rwanda, previous digital health interventions such as Rapid SMS, were utilized in community and health centers to enhance maternal and child health [29]. The successful implementation of these interventions served as a positive indicator, suggesting that the introduction of e-tracker would also be successful.

Respondents acknowledged the notable improvements in data access, quality, management, and security brought about by the introduction of e-tracker systems and other digital health interventions. Similar findings, where digital health interventions were recognized for improving data quality and management, have been demonstrated in previous studies [30,31,32]. Digital health interventions in Rwanda benefited both the providers and patients by reducing the time they spend at the health center for vaccinations. Additionally, the e-tracker simplified the verification process by providing individualized data, allowing healthcare providers to ensure the accuracy of vaccination records, especially when individuals required proof of vaccination. This contrasts with other studies where digital health interventions were only embraced by patients, while healthcare providers perceived them as an additional workload given to them and a means of exercising internal hospital operations and oversight control rather than a tool to improve patient care [31, 33].

While direct causation between the HPV e-tracker system and increased vaccination coverage remains inconclusive, its influence on follow-up activities is evident. The respondents agreed that the digital systems aided in effective follow-ups, reducing missed appointments, and enabling healthcare providers to track the locations of the second HPV vaccination dose, even when individuals changed vaccination sites. These findings align with those from previous studies that reported improved patient follow-up and healthcare delivery due in part to digital health interventions [30, 34]. In Kenya, it was observed that improved healthcare delivery could not be solely attributed to data quality from the use of digital health interventions as a stand-alone factor; other contributors included incentives and motivations provided to users [35].

Despite the system’s benefits, some challenges persist, particularly related to the end-user's internet connectivity. The problem of the internet hindering the implementation of digital health interventions has also been reported in other LMICs [34, 36, 37]. Hence, evaluating and ensuring the availability of a stable internet connection as part of the overall preparation before introducing digital health interventions is essential for implementation in LMICs [13, 37,38,39]. Additionally, issues arise in data management, such as challenges with data entry personnel for overloaded sites, problems with duplicate entries and issues related to re-registration. Operational challenges including lack of interoperability when more than one system is in use, have also been reported. Similar challenges have been observed in other settings [40]. To overcome these challenges, a problem-centered approach involving training provision and the establishment of a stable evaluation framework have been advocated [40].

Nurses, as primary end-users, initially faced hurdles due to limited computer literacy. However, with time, they adapted and recognized the system’s advantages over the traditional paper-based method. Computer literacy has been cited as a challenge in implementing digital health interventions in other settings [37, 39, 41]. Other barriers to implementation of digital health interventions reported in other studies include lack of interoperability, start-up and maintenance costs, high turnover of the healthcare providers, and lack of user-friendly design [33, 40,41,42,43].

End-user feedback emphasized the importance of a stable internet connection and adequate equipment. The rapid implementation of the e-tracker for HPV vaccinations was facilitated by the availability of tablets and internet connectivity, highlighting the importance of resource availability for successful implementation [37, 40].

Leadership commitment played a crucial role in the successful transition from paper-based systems to digital systems. Regular routine monitoring, combined with round-the-clock assistance from the central level and collaboration with health facilities, ensured data completeness. Similar factors facilitating the implementation of digital health interventions have been described in other studies, and lack of leadership support were a cited reason for failed implementation [44, 45]. Lack of integrating digital health interventions into the health care system was reported as a demotivating factor for implementation [33] The collaborative approach in Rwanda promoted continuous improvement and streamlined the integration process. The political will and leadership commitment involve allocating financial resources, setting policies and regulations, and gathering support from various stakeholders to ensure the successful integration of the digital health interventions. And these are essential in overcoming barriers such as resistance to change and other competing priorities [46].

This qualitative study is unique in the diverse composition of participants, which includes various perspectives and experiences from individuals involved at different levels within Rwanda’s healthcare system. Moreover, the study thoroughly examined a broad spectrum of digital health interventions implemented across different tiers of the healthcare system in Rwanda. Consequently, this approach provided a comprehensive understanding of the utilization of these digital health interventions.

The research team took extensive measures to minimize bias and ensure integrity of the study including training the data collectors, creating a safe space for interviews, and anonymizing responses.

The study limitations include a small sample size of 15 participants, selected through purposeful sampling. Although efforts were made to ensure data saturation and maximum variation, the findings may not fully capture the diversity of experiences and perspectives among all users of digital health interventions for cervical cancer services in Rwanda. Social desirability bias may have influenced participants’ responses, potentially affecting the accuracy of their feedback on the usability and effectiveness of these interventions. In addition, successful implementation and integration may rely on access to reliable technology infrastructure and resources, which can vary across different healthcare facilities and regions. Finally, the study was conducted within a specific time frame, potentially limiting the exploration of evolving user experiences over time. The long-term sustainability and scalability of digital health interventions beyond the study period should be considered.

Conclusion

In conclusion, the implementation of the e-tracker, mUzima and OpenMRS systems in Rwanda HPV vaccination and cervical cancer services indicates a significant step toward efficient data management and improved healthcare delivery. While some challenges persist, the successful adaptation of healthcare providers and the active support from leadership highlight the potential for further advancements in digital health interventions in Rwanda.

We recommend continued leadership support and collaboration at all levels of the healthcare system, along with regular monitoring and assistance from the central level, to ensure the sustained success of digital health interventions. Additionally, implementing a continuous evaluation framework that allows learning from both successes and challenges, adapting strategies as needed, and addressing and strengthening digital health infrastructure will facilitate efficient data management and enhance healthcare delivery. In a country like Rwanda, where digital health initiatives are emerging at a high rate, there is a crucial need for policy development with clear guidelines and regulations to ensure the promotion of interoperability and ethical use of digital health interventions, safeguarding patient privacy and data security.