Objectives
This study aimed to develop evaluation criteria for the elementary-school-based health promotion program using the RE-AIM framework and to examine their feasibility.
Methods
Previous evaluation studies on health interventions for elementary-school students using the RE-AIM framework were reviewed systematically to identify appropriate evaluation criteria. A diet and physical activity intervention based on the transtheoretical model was implemented in a pilot study using the “Happy Me” application. The feasibility of using the RE-AIM framework to evaluate it was examined.
Results
The review yielded the following evaluation criteria: “reach,” the ratio of participants out of the total target population; “efficacy/effectiveness,” the difference in outcomes between the intervention and control groups, or between a pre- and post-test; “adoption,” the rate of use of the program and participation in the next stage of the program; “implementation,” the progress on the program components; “maintenance,” the participants’ and teachers’ intention to continue using the program. The pilot study reached 76.6% of the targeted population. The intake of sugar-sweetened beverages decreased (P < 0.0001), and the duration of walking increased (P < 0.0001). Other indicators could not be evaluated; therefore, potential indicators were suggested.
Conclusions
This study produced feasible evaluation criteria for elementary-school-based health promotion using the RE-AIM framework. Nevertheless, the feasibility needs to be validated with a broader range of studies and long-term interventions.
This study aimed to develop evaluation criteria for the elementary-school-based health promotion program using the RE-AIM framework and to examine their feasibility.
Previous evaluation studies on health interventions for elementary-school students using the RE-AIM framework were reviewed systematically to identify appropriate evaluation criteria. A diet and physical activity intervention based on the transtheoretical model was implemented in a pilot study using the “Happy Me” application. The feasibility of using the RE-AIM framework to evaluate it was examined.
The review yielded the following evaluation criteria: “reach,” the ratio of participants out of the total target population; “efficacy/effectiveness,” the difference in outcomes between the intervention and control groups, or between a pre- and post-test; “adoption,” the rate of use of the program and participation in the next stage of the program; “implementation,” the progress on the program components; “maintenance,” the participants’ and teachers’ intention to continue using the program. The pilot study reached 76.6% of the targeted population. The intake of sugar-sweetened beverages decreased (P < 0.0001), and the duration of walking increased (P < 0.0001). Other indicators could not be evaluated; therefore, potential indicators were suggested.
This study produced feasible evaluation criteria for elementary-school-based health promotion using the RE-AIM framework. Nevertheless, the feasibility needs to be validated with a broader range of studies and long-term interventions.
Childhood obesity is a health problem that cannot be overlooked because it is increasing worldwide [1]. In 2013, almost a quarter of children in developed countries and more than 10% in developing countries were overweight or obese based on the body mass index (BMI) [2]. Children with obesity are likely to remain obese in adulthood [3, 4]. They are more at risk of developing adult health problems, such as type 2 diabetes, heart disease, stroke, various types of cancer, and osteoarthritis [5, 6, 7]. The dietary and physical activity behaviors of children are affected by many social factors, and schools play a vital role by establishing a secure and cooperative environment with policies and practices to support healthy behaviors [8].
To carry out an intervention program effectively, there is a need for a systematic and scientific method to evaluate the effects of the program objectively on various levels [9, 10]. The RE-AIM framework developed by Glasgow et al. has been used to assess the overall health impact of health promotion programs [11, 12]. The RE-AIM framework aims to assess the effectiveness of the health promotion program and seek more efficient resource allocation in the future with reference to five dimensions: reach, efficacy, adoption, implementation, and maintenance [11]. According to a systematic review [13] of interventions focused on applying the RE-AIM framework, 26 out of the 71 articles published from 1999 to December 2010 pertained to physical activity and obesity; only two were conducted in the school setting. Thus, few studies on physical activity and nutrition intervention in a school setting have been evaluated using the RE-AIM framework.
Mobile phones and the transtheoretical model (TTM) allow for more systematic and effective implementation of intervention programs. Intervention programs using mobile phones have the advantage that service users can easily receive tailored information anytime, anywhere [14]. Mobile-based approaches are beneficial for implementing successful interventions on eating habits and physical activity [15]. The TTM, developed by Prochaska et al. [16], has been used in intervention studies to assess various behavioral changes, enabling tailored education at the stage of behavioral change [17]. Mobile technology or TTM have been applied to various interventions in adults, but limited studies have used the same with children [18]. In addition, few studies have applied new technology and the TTM simultaneously [19].
This study developed feasible evaluation criteria of the RE-AIM framework for application to an elementary-school-based health promotion program. First, potential evaluation criteria were proposed with an analysis of previous studies that evaluated the elementary-school-based health promotion program using the RE-AIM framework. Second, the feasibility of the RE-AIM framework was evaluated using the authors’ elementary-school-based pilot health promotion program.
Domestic and international studies were searched to examine the application of the RE-AIM framework for evaluating health promotion programs. Fig. 1 presents the selection process. A literature search was conducted using four databases (RISS, Koreamed, PubMed, and ScienceDirect), using the following search keywords: child, nutrition education, physical activity education, health intervention, nutrition intervention, diet intervention, and physical activity intervention. The literature search process was iterative and was conducted using various combinations of keywords, using the “AND” function in the search options.
Fig. 1
Flow diagram of the selection process of intervention studies
Thus, 120 studies were identified from international databases. No domestic studies evaluating the effects of nutrition, physical activity, and eating habits on elementary-school children using the RE-AIM framework were found. After excluding duplicate studies, 62 studies were selected initially. The abstracts of the selected studies were screened, and the final four intervention studies on elementary school children were selected (Fig. 1).
A pilot study was conducted to assess the feasibility of an elementary-school-based health promotion program. The pilot study was conducted as a non-randomized controlled trial of an intervention on a healthy diet and physical activity using the “Happy Me” mobile application developed as a tailored mobile service for the prevention and management of childhood obesity [20, 21]. The subjects were 5th to 6th-grade students from four elementary schools in Gimpo City and Incheon City. The four research schools were assigned as representative schools in each region by the Gimpo Office of Education Support and the Incheon Metropolitan City Office of Education. Parents of the participating students received an informed consent form explaining the nature and procedure of the study. The study was approved by the Inje University Seoul Paik Hospital Institutional Review Board (IIT-2015-070). Of the 989 students, 166 who did not provide written informed consent to participate in the study and 37 who did not participate in the pre- or post-tests were excluded. Of the 786 participants remaining, 558 students used the mobile application aimed at obesity prevention and management (intervention group), whereas 228 students followed the regular curriculum (control group) for 12 weeks from August 20, 2015, to November 20, 2015. Both groups received one session of offline education on the nutrition and physical activity. The measures used in the pre- and post-tests of the study included eating habits (i.e., frequency of intake of breakfast, vegetables, fruit, sugar-sweetened beverages, fast food, instant food, snacks, and late-night snacks), physical activity (i.e., frequency of engaging in physical activities that make it difficult to breathe, intense physical activities, mild physical activities, and strengthening exercise per day; the number of days on which one walked for more than 10 minutes (per week); the duration of walking in a day), screen time, physical measurements (i.e., height, weight, BMI, waist circumference, and hip circumference), and results of the Physical Activity Promotion System (PAPS; i.e., cardiopulmonary endurance, flexibility, power, and strength).
Four studies [22, 23, 24, 25] were selected based on the screening criteria (Table 1). Two studies involved physical activity intervention [22, 23], and two involved nutrition intervention [24, 25] for elementary school students aged 6 ~ 12 years. The sample sizes ranged from 1,147 to 2,280. There were three randomized controlled trials (RCT) with parallel groups [23, 24, 25] and one cluster randomized controlled trial (CRCT) [22]. In two studies, two different samples were used: the dissemination sample (to evaluate Reach, Adoption, and Maintenance) and the program evaluation sample (to assess Efficacy/Effectiveness and Implementation) [24, 25]. The dissemination sample consisted of all classrooms that ordered the program and the students who participated in the program. The program evaluation sample for the RCT was recruited from local public elementary schools. Table 2 summarizes the results of evaluating these studies using the RE-AIM framework.
Table 1
Summary of the selected studies
Table 2
Evaluation results of the selected studies using the RE-AIM framework
This review was conducted on a specific target population (i.e., elementary-school students). The baseline sample sizes of the reviewed studies ranged from 1,147 to 2,280 (median = 1,837.5) participants. All studies reported the age (range = 6 to 12 years) and sex of the participants. The participation rate was reported in all studies, and it ranged from 33% to 99.9%.
All reviewed studies included measures of nutrition knowledge, self-efficacy, outcome expectations, intake of foods, or outcomes related to physical activity. The nutrition knowledge variables (reported knowledge about food groups, main nutrients, nutrient functions, breakfast choices, snack choices, and dinner choices) were measured in two studies [24, 25]. Both reported statistically significant improvements in the intervention group with reference to all nutrition knowledge variables compared to the control group. Self-efficacy and outcome expectations were measured in the two studies [24, 25], and both reported significant improvements in the intervention group compared with the control group. The intake of foods was measured in two studies [24, 25], and both reported statistically significant improvements in some food groups (i.e., vegetables, low-nutrient high-energy foods, sugar-sweetened beverages) in the intervention group as compared with the control group. The outcomes related to physical activity were measured in the two studies [22, 23], one of which reported significant improvements in energy expenditure during recess in the intervention group compared to the control group [23].
At the organizational level, all the studies specified the location of the intervention as the school. The rate at which the schools adopted the intervention program was reported in all studies, ranging from 9% to 66.7%. The “Adoption” rate was calculated as the rate of schools that agreed to participate in the program [22, 23] or the rate at which the classrooms ordered the program [24, 25].
All the reviewed studies described the intervention duration, which ranged from four weeks to two school years. Fidelity of implementation was reported as follows: 96% of the teachers distributed all newsletters; 71% of the teachers taught the exercises most of the time; 28% of the students and 55% of the parents had read all the newsletters [22]; 71.4% of the program elements were implemented successfully [23]; 100% or 88.9% of the teachers in the evaluation sample implemented all the lessons [24, 25].
All reviewed studies described their maintenance standards. Three of the studies examined whether the teachers wanted the program to be a standard practice in their curriculum or whether they wanted to re-order the program materials during the subsequent school year [22, 23, 24, 25]. The other study examined how well the program elements were maintained at the 18-month follow-up [23].
Before evaluating the feasibility of the RE-AIM framework, criteria for evaluating the intervention with a mobile application and the TTM were developed (Table 3). The feasibility of this evaluation system using the RE-AIM framework was assessed by evaluating the individual level of “reach” and “efficacy/effectiveness” of a pilot study involving intervention in healthy eating habits and physical activity (Table 4).
Table 3
Suggested evaluation criteria in each dimension of the RE-AIM framework for the intervention programs using a mobile application based on the transtheoretical model
Table 4
Evaluation of the impact of the intervention using the mobile application “Happy Me” using the RE-AIM framework
The schools (n = 4) invited to participate in the pilot study consisted of 989 students in Grades 5 and 6. The study population consisted of 786 students at the start; 759 students completed the post-test after the 12-week intervention. The “Reach” of the target population (four elementary schools in Gimpo City and Incheon City) was 76.7%.
Efficacy/Effectiveness was assessed through pre- and post-tests on the eating habits, physical activity, screen time, physical measurements, and the PAPS. Among the changes in food intake frequency, in the intervention group, the intake frequency of sugar-sweetened beverages decreased from 2.3 ± 2.6 times/week to 1.6 ± 1.5 times/week after intervention (P < 0.0001). The magnitude of the effect was larger in the intervention group than in the control group (P < 0.0286). The duration of walking increased from 74.2 ± 89.9 minutes/day to 95.2 ± 105.1 minutes/day in the intervention group, owing to the changes in exercise time (P < 0.0001). The increase in walking duration in a day was larger in the intervention group than the control group (P = 0.0064). Such changes were not observed in the control group. Changes in the TTM stages and self-efficacy were expected to be evaluated by the item, “Are you confident in your ability to accept the challenge” and “change in the TTM stages related to food intake and physical activity” according to the mobile application used. However, these data had a large number of instances of selection of food groups (i.e., vegetables, fruits, sugar-sweetened beverages, fast food, instant food, snacks, and late-night snacks) and acceptance of challenges. Additional reference criteria for the evaluation, such as the most important challenge and the challenge with which the participant is most confident, are required to reflect the “Efficacy” of the intervention. In addition, the changes in knowledge, perception, and attitudes regarding healthy eating habits and physical activities, and availability of and satisfaction with the application were suggested to be included in the evaluation criteria in a future study.
These dimensions of the RE-AIM framework did not apply to the measurement tools of the pilot study. Therefore, the following potential indicators are suggested for use in future studies: “Adoption” could be assessed based on the proportion of schools or students who are willing to continue using the “Happy Me” application in the future, the proportion of students who have gained knowledge and have achieved their goals through the application could be used as an indicator of “Implementation,” and “Maintenance” could be assessed based on the proportion of students who exhibited a positive change in their food intake and health condition after six months.
This study aimed to identify evaluation criteria used in the RE-AIM framework using a systematic review and evaluated the feasibility of the RE-AIM framework using an elementary-school-based pilot health promotion program. The systematic review revealed no studies in Korea evaluating the effectiveness of health promotion programs for elementary-school students using the RE-AIM framework. Thus, four international studies were selected [22, 23, 24, 25]. In the four studies selected, “Reach” was assessed by the rate of participation in the program out of the eligible population. “Efficacy” was assessed based on the pre- and post-test outcomes for the intervention group and control groups. “Adoption” was assessed by the usage rate of the program and the participation rate in the next stage of the program; “Implementation” was assessed by the progress of the program components and lessons. “Maintenance” was assessed by the rate of participants and teachers who responded positively to the program and intended to continue to use it. The feasibility of the RE-AIM framework was examined by applying it to evaluate a pilot study based on the evaluation components identified from the review. Students (n = 989) from Grades 5 and 6 from four elementary schools in Gimpo City and Incheon City were intervened for 12 weeks using the “Happy Me” application developed as a customized mobile service for the prevention and management of childhood obesity. The results of the intervention were evaluated using the RE-AIM framework. Reach and Efficacy could be evaluated, but Adoption, Implementation, and Maintenance could not be assessed in the pilot study. Accordingly, assessment indicators, such as future participation rate, knowledge improvement rate, and availability of or satisfaction with the application, are suggested for future studies.
Previous studies used the RE-AIM framework in the evaluation phase, planning the intervention, and reviewing the literature on health intervention [9, 26, 27]. On the other hand, only one Korean study that used the RE-AIM framework to evaluate an alcohol education program for university students was found [28]. In this study, the RE-AIM framework was used to develop an evaluation plan in the development stage. Reach was assessed with reference to the rate of students who had received alcohol education. Efficacy was evaluated by the changes in awareness, knowledge, attitude, and skills pertaining to moderate drinking. Adoption was analyzed by the rate of schools that adopted the alcohol education program. Implementation was examined by the degree of change in the educated students’ knowledge and utilization of skills and whether the staff followed the instructional guidelines. Maintenance was determined by whether the students’ moderate drinking practice lasted longer than six months and whether alcohol education was included as a regular component of the curriculum.
In previous studies [22, 23, 24, 25], the RE-AIM framework was used to evaluate the effects of the nutrition and physical activity intervention in elementary-school students in the United States and the Netherlands. On the other hand, in the present study, the pilot study involved sending messages using a mobile application as an intervention based on the TTM. Mobile intervention is highly accessible and cost-effective and can provide personalized information [29, 30, 31, 32, 33, 34]. Additionally, the TTM is the most promising model for promoting behavioral changes associated with healthy lifestyle habits [35].
Similar to the intervention method used in this study, three studies used a mobile application and the TTM simultaneously [36, 37, 38]. One of these, in a study conducted in Australia, messages developed using the TTM were sent to participants for 12 weeks via their mobile phones [38]. The study was similar to the present study in the contents and intervention period. The intervention group received eight mobile messages, one email, five personal coaching calls, a dietary brochure, and other materials through the mobile application and website every week. By contrast, the control group received four mobile messages and a printed dietary and physical activity guideline only. The pre- and post-online surveys showed that after 12 weeks, the intervention group had a lower weight, higher daily intake of vegetables, and lower intake of sweet beverages and instant food than the control group. Furthermore, the total amount of physical activity in the intervention group increased by 252.2 MET-minutes per week, and the frequency increased by 1.3 days per week. Patridge et al.[38] reported a greater magnitude of changes in the intake frequency of vegetables and instant food than those observed in the present study. These differences are because the target population and the intervention factors in the two studies were different. The target population of the present study was 5th and 6th-grade elementary school students, while that by Patridge et al.[38] was 18- to 35-year-old young adults with a high risk of weight gain and overweight. In addition to providing messages using the mobile application, Patridge et al. also provided information through personal coaching calls and a dietary brochure [38].
The limitations of the present study are as follows. First, the mobile intervention had several advantages, but it was difficult to assess the Maintenance aspect because a longer intervention period indicated a lower participation rate [39]. Second, although the 12-week period was sufficient for obesity intervention, further research is required because several school-based prevention and management programs demand long-term intervention [40]. Third, because this study evaluated the results of the pilot study using the RE-AIM framework, it is necessary to reevaluate the same in an expanded study in the future. Fourth, the pilot study did not include a dietary intake measurement tool in the pre- and post-test. To evaluate the efficacy in future studies, the dietary intake needs to be assessed in detail using methods such as the 24-hour recall. Nevertheless, this study was meaningful because it was the first to evaluate the feasibility of the RE-AIM framework through a systematic review of elementary-school-based intervention studies. In addition, the effects of intervention with a mobile application and TTM were similar to those of previous studies, suggesting that these methods have the potential for effective intervention.
In this study, only some of the five dimensions of the RE-AIM framework were evaluated because of the limitations of the measurement tools. According to a systemic review of the RE-AIM framework, most studies using the RE-AIM framework indicated Reach and Efficacy. On the other hand, only a few addressed Adoption, Implementation, and Maintenance. In the future, more effective program evaluation and dissemination will be possible if the evaluation criteria are selected by considering all five dimensions of the RE-AIM framework at the planning stage of the intervention program.
This work was supported by the Bio & Medical Technology Development Program of the National Research Foundation (civil research project for solving social problems through the (NRF)) and funded by the Ministry of Science, ICT & Future Planning (Grant No. 2013M3C8A2A01078464, 2014M3C8A2A01070970, NRF-2020R1A2C1011675).