{rfName}
As

Indexed in

License and use

Icono OpenAccess

Citations

9

Altmetrics

Analysis of institutional authors

Grau-Corral ICorresponding AuthorKostov BAuthor Puig-Soler MAuthorRoca DAuthorSiso-Almirall AAuthor

Share

August 4, 2021
Publications
>
Article
Sí

Assessing Apps for Health Care Workers Using the ISYScore-Pro Scale: Development and Validation Study

Publicated to:Jmir Mhealth And Uhealth. 9 (7): e17660- - 2021-07-01 9(7), DOI: 10.2196/17660

Authors: Grau-Corral, Inmaculada; Efrain Pantoja, Percy; Grajales, Francisco J, III; Kostov, Belchin; Aragunde, Valentin; Puig-Soler, Marta; Roca, Daria; Couto, Elvira; Siso-Almirall, Antoni

Affiliations

Biomed Res Inst St Pau IRB St Pau, Iberoamer Cochrane Ctr, Barcelona, Spain - Author
Consorci Atencio Primaria Salut Barcelona Esquer, Casanova Primary Hlth Care Ctr, Barcelona, Spain - Author
Consorci Atencio Primaria Salut Barcelona Esquer, Corts Primary Hlth Care Ctr, Barcelona, Spain - Author
Fdn iSYS Internet Salud & Soc, C Mallorca 140 2-4, Barcelona 08036, Spain - Author
Fundacion iSYS - Author
Hosp Clin Barcelona, Barcelona, Spain - Author
Hospital Clinic Barcelona - Author
Inst Investigac Biomed August Pi & Sunyer IDIBAP, Primary Healthcare Transversal Res Grp, Barcelona, Spain - Author
Institut d'Investigacions Biomèdiques August Pi i Sunyer - IDIBAPS - Author
Institut dInvestigacio Biomedica Sant Pau (IIB Sant Pau) - Author
Les Corts Primary Healthcare Center - Author
Primary Care Centre Casanova - Author
The University of British Columbia - Author
Univ Barcelona, Barcelona, Spain - Author
Univ British Columbia, Vancouver, BC, Canada - Author
Universitat de Barcelona - Author
See more

Abstract

Background: The presence of mobile phone and smart devices has allowed for the use of mobile apps to support patient care. However, there is a paucity in our knowledge regarding recommendations for mobile apps specific to health care professionals. Objective: The aim of this study is to establish a validated instrument to assess mobile apps for health care providers and health systems. Our objective is to create and validate a tool that evaluates mobile health apps aimed at health care professionals based on a trust, utility, and interest scale. Methods: A five-step methodology framework guided our approach. The first step consisted of building a scale to evaluate apps for health care professionals based on a literature review. This was followed with expert panel validation through a Delphi method of (rated) web-based questionnaires to empirically evaluate the inclusion and weight of the indicators identified through the literature review. Repeated iterations were followed until a consensus greater than 75% was reached. The scale was then tested using a pilot to assess reliability. Interrater agreement of the pilot was measured using a weighted Cohen kappa. Results: Using a literature review, a first draft of the scale was developed. This was followed with two Delphi rounds between the local research group and an external panel of experts. After consensus was reached, the resulting ISYScore-Pro 17-item scale was tested. A total of 280 apps were originally identified for potential testing (140 iOS apps and 140 Android apps). These were categorized using International Statistical Classification of Diseases, Tenth Revision. Once duplicates were removed and they were downloaded to confirm their specificity to the target audience (ie, health care professionals), 66 remained. Of these, only 18 met the final criteria for inclusion in validating the ISYScore-Pro scale (interrator reliabilty 92.2%; kappa 0.840, 95% CI 0.834-0.847; P<.001). Conclusions: We have developed a reproducible methodology to objectively evaluate mobile health apps targeted to health care professionals and providers, the ISYScore-Pro scale. Future research will be needed to adapt the scale to other languages and across other domains (eg, legal compliance or security).

Keywords

cancer survivorsconsensushealth care professionalsmhealthmobile appmobile applicationmobile application rating scalescale developmenttechnologyAssessmentHealth care professionalsHealth personnelHumansMhealthMobile appMobile applicationMobile application rating scaleMobile applicationsReproducibility of resultsScale developmentSurveys and questionnairesUser acceptance

Quality index

Bibliometric impact. Analysis of the contribution and dissemination channel

The work has been published in the journal Jmir Mhealth And Uhealth due to its progression and the good impact it has achieved in recent years, according to the agency WoS (JCR), it has become a reference in its field. In the year of publication of the work, 2021, it was in position 21/109, thus managing to position itself as a Q1 (Primer Cuartil), in the category Health Care Sciences & Services.

From a relative perspective, and based on the normalized impact indicator calculated from the Field Citation Ratio (FCR) of the Dimensions source, it yields a value of: 1.32, which indicates that, compared to works in the same discipline and in the same year of publication, it ranks as a work cited above average. (source consulted: Dimensions Aug 2025)

Specifically, and according to different indexing agencies, this work has accumulated citations as of 2025-08-07, the following number of citations:

  • WoS: 3
  • Scopus: 3
  • Europe PMC: 3

Impact and social visibility

From the perspective of influence or social adoption, and based on metrics associated with mentions and interactions provided by agencies specializing in calculating the so-called "Alternative or Social Metrics," we can highlight as of 2025-08-07:

  • The use, from an academic perspective evidenced by the Altmetric agency indicator referring to aggregations made by the personal bibliographic manager Mendeley, gives us a total of: 45.
  • The use of this contribution in bookmarks, code forks, additions to favorite lists for recurrent reading, as well as general views, indicates that someone is using the publication as a basis for their current work. This may be a notable indicator of future more formal and academic citations. This claim is supported by the result of the "Capture" indicator, which yields a total of: 46 (PlumX).

With a more dissemination-oriented intent and targeting more general audiences, we can observe other more global scores such as:

  • The Total Score from Altmetric: 12.35.
  • The number of mentions on the social network X (formerly Twitter): 21 (Altmetric).

It is essential to present evidence supporting full alignment with institutional principles and guidelines on Open Science and the Conservation and Dissemination of Intellectual Heritage. A clear example of this is:

  • The work has been submitted to a journal whose editorial policy allows open Open Access publication.

Leadership analysis of institutional authors

This work has been carried out with international collaboration, specifically with researchers from: Canada.

There is a significant leadership presence as some of the institution’s authors appear as the first or last signer, detailed as follows: First Author (Grau Corral, Inmaculada) and Last Author (Sisó Almirall, Antoni).

the author responsible for correspondence tasks has been Grau Corral, Inmaculada.