Pervasive and Mobile Computing (
`
`)
`
`–
`
`Contents lists available at SciVerse ScienceDirect
`
`Pervasive and Mobile Computing
`
`journal homepage: www.elsevier.com/locate/pmc
`
`The mobile fitness coach: Towards individualized skill assessment using
`personalized mobile devices
`Matthias Kranz a, Andreas Möller b,∗, Nils Hammerla c, Stefan Diewald b, Thomas Plötz c,
`Patrick Olivier c, Luis Roalter b
`a Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering, Luleå, Sweden
`b Technische Universität München, Distributed Multimodal Information Processing Group, Munich, Germany
`c Newcastle University, Culture Lab, School of Computing Science, Newcastle upon Tyne, United Kingdom
`
`a r t i c l e
`
`i n f o
`
`a b s t r a c t
`
`Article history:
`Available online xxxx
`
`Keywords:
`Skill assessment
`Physical exercising
`Mobile computing
`Mobile HCI
`Human-computer interaction
`Sports
`Activity recognition
`Pervasive computing
`Ubiquitous computing
`
`1. Introduction
`
`We report on our extended research on GymSkill, a smartphone system for comprehensive
`physical exercising support, from sensor data logging, activity recognition to on-top
`skill assessment, using the phone’s built-in sensors. In two iterations, we used principal
`component breakdown analysis (PCBA) and criteria-based scores for individualized and
`personalized automated feedback on the phone, with the goal to track training quality and
`success and give feedback to the user, as well as to engage and motivate regular exercising.
`Qualitative feedback on the system was collected in a user study, and the system showed
`good evaluation results in an evaluation against manual expert assessments of video-
`recorded trainings.
`
`© 2012 Elsevier B.V. All rights reserved.
`
`Regular physical activity is important to personal health and well-being, both for the individual and the society.
`Encouraging people to exercise more is key to maintaining or regaining personal health but, unfortunately, difficult to
`achieve in practice. One barrier to exercise is that lay people often are insufficiently knowledgeable about effective and
`safe physical exercises. Maintaining an exercise regime over longer periods of time requires high levels of motivation. It is
`well established that access to a personal trainer has a significant impact on both adherence and motivation to a program
`of physical exercise [1], and the quality of the exercise undertaken [2]. They continuously monitor the exercises and both
`provide individualized advice and motivate the trainee. Personal trainers also play an important role in rehabilitation, e.g.,
`exercise programs for muscle recovery after surgery, for which there is a need for advice regarding effectiveness and safety.
`Unfortunately, this is too expensive to provide over extended periods of time, and where financial factors are not a barrier,
`personal privacy preferences can be (i.e., the perception of the potential for embarrassment).
`Smartphones, being pervasive devices, are ideal to support and contribute to regular physical exercising. Apps for all
`purposes have turned the phone to a multi-functional device, far beyond its classic domain of application. They transform
`the phone to a platform for a variety of applications pervading everyday life: From reading news on the go, over location-
`based services, games, up to specialized leisure-time and ‘hobby’ apps for every flavor. Musicians can turn their phone into
`a guitar tuner, gourmets into a wine guide, and so on. This trend is also visible in the sports domain. Increasing processing
`
`∗ Corresponding author.
`
`E-mail addresses: matthias.kranz@ltu.se (M. Kranz), andreas.moeller@tum.de (A. Möller), nils.hammerla@newcastle.ac.uk (N. Hammerla),
`stefan.diewald@tum.de (S. Diewald), thomas.ploetz@newcastle.ac.uk (T. Plötz), patrick.olivier@newcastle.ac.uk (P. Olivier), roalter@tum.de (L. Roalter).
`
`1574-1192/$ – see front matter © 2012 Elsevier B.V. All rights reserved.
`doi:10.1016/j.pmcj.2012.06.002
`
`Petitioner Apple Inc. – Ex. 1020, p. 1
`
`

`

`2
`
`M. Kranz et al. / Pervasive and Mobile Computing (
`
`)
`
`–
`
`power, integrated sensors, and the ability for rich, multimodal interaction do qualify smartphones not only as personal
`assistants, but also as personal fitness coaches for supporting individualized training and skill assessment.
`The current generation of smartphones with their wealth of sensing, communication, and computing capabilities
`represents an ideal platform for replicating the services of personal trainers in a way that is accessible to and affordable for
`virtually everyone. Enabling expert assessment of physical exercises and/or rehabilitation in everyday life (sports) situations
`integrates mobile interaction further into the real world and has the potential to have a substantial positive impact on
`people’s lives.
`We report on our research on an integrated human-activity assessment system, which supports lay users in their
`everyday life exercising routine with automatically generated expert assessments. The presented system captures the
`complete training process from exercise descriptions, sensor data logging, activity recognition and on-top skill assessment.
`This allows the user for individualized, self-determined training supported by coach-quality feedback. Our approach uses
`balance board training as one representative example for physical exercise support, applicable to a wide range of users
`and scenarios, both for fitness training and rehabilitation. In an iterative process, we developed a smartphone application
`that uses integrated sensors for exercise skill assessment on multiple levels for a variety of exercises. We understand skill
`assessment as individualized, personalized and automated feedback that allows tracking training quality and success, with
`the claim to replicate the expert judgments of a professional human coach.
`As a basis for our research, we performed an extensive review of both commercial applications and scientific research
`on training support. With the assistance of a sports medicine specialist, we developed an exercise set, exercise descriptions
`and training plans that we evaluated in an iterative design process with a focus group. Six users performed a 20-exercise set
`twice a day during one week and produced 1200 exercise records. This basis was manually assessed by an expert and used
`as a basis and evaluation of ground truths of our assessment algorithms.
`In a first prototype, a novel unsupervised and automated activity analysis approach – principal component breakdown
`analysis (PCBA) – is used to generate informative and easy to interpret qualitative user feedback in the form of color maps.
`Based on this, a second iteration then goes further and generates more precise and targeted feedback on individual qualities
`of the performed exercises directly on the device.
`Our assessment algorithms show high accuracy in practical evaluations compared to manual judgments provided by
`sports medicine experts. Qualitative user feedback indicated the acceptance of our system and a notable potential for
`maintaining long-term exercising engagement. The analysis provided to the user is consistent to the professional analysis
`of a personal coach. This was confirmed in the two studies presented with the aid of an external expert. Physical interaction
`with training devices is facilitated using NFC-augmented (Near Field Communication) balance boards.
`We show for the first time that an automated approach utilizing sensing and computing capabilities of modern mobile
`phones can provide expert-level feedback on the quality of physical exercises as they are regularly performed for recreation
`or rehabilitation purposes. This approach has great potential for the generalizability of physical mobile interaction in sports,
`especially in the context of the aging society, but also in the more general domain of human skill assessment of everyday
`activities, which is relevant for a number of therapeutical applications, e.g. skill maintenance in Dementia or Parkinson’s
`care. Our evaluation indicates that we can keep the motivation with coach-like feedback longer than with conventional
`approaches.
`
`2. State of the art
`
`Advances in the miniaturization of (inertial) sensing technology, along with the increasing availability of smartphones
`as a sensing platform, have given rise to many commercial appliances, apps, and general academic interest into people’s
`physical activity. This allows for embedded interaction [3], that is to look at new opportunities and application areas that
`arise for interactive systems and at novel ways for human–computer interaction (HCI) enabled thereby.
`Beyond straightforward applications in medicine, where physical activity has to be quantified reliably, many lifestyle,
`sports and professional assessment systems have been developed. This section first explores systems that employ inertial
`sensors to assess different aspects of physical activity. Subsequently, the landscape of (commercial and free) health and
`fitness applications on smartphones is summarized in a comparative study.
`
`2.1. Automatic assessment of physical activity
`
`Body-worn and pervasive sensors have been employed in a large variety of recognition scenarios, identifying different
`types of physical activities [4–8] with satisfying accuracy. Quantifying qualitative aspects of human motion, such as motor
`performance, has been of intense focus in medicine, where particularly the assessment of degenerative conditions such as
`Parkinson’s disease is of interest [9,10]. Automated skill assessment for domains where less prior knowledge is accessible is
`a relatively novel application of pervasive computing. Here augmentation of physical training devices with sensors has been
`used as the basis for monitoring outdoor sports such as skiing [11] or tennis [12], as well as indoors, such as recognition
`and tracking of free-weight exercises with accelerometers in a glove [13]. In the gym, sensor data from balance board
`training [14,15] has been used to provide feedback on the performance quality. Furthermore, body-worn inertial sensors
`have been used for the assessment of (professional) athletes in sports such as snow-boarding [16], swimming [17] and
`
`Petitioner Apple Inc. – Ex. 1020, p. 2
`
`

`

`M. Kranz et al. / Pervasive and Mobile Computing (
`
`)
`
`–
`
`3
`
`Table 1
`The 15 health and fitness apps considered by this review can be classified into the categories GPS tracker, workout planner and exercise book. The last two
`columns indicate the number of reviews and average ratings in the Android market.
`Category
`Name
`Developer
`GPS tracker
`AndAndo
`Javi Pacheco
`GPS tracker
`Cardio trainer
`WorkSmart labs inc.
`GPS tracker
`miCoach
`Adidas
`GPS tracker
`RunKeeper
`Fitness keeper inc.
`GPS tracker
`runtastic
`runtastic
`GPS tracker
`Sports tracker
`Endomondo
`GPS tracker
`Sports tracker
`Sports tracking ltd.
`Workout planner
`Ab workout free
`Daniel Miller
`Workout planner
`Abs & core
`PumpOne
`Workout planner
`Body fitness
`Health team
`Workout planner
`C25K lite
`Guy Hoffmann
`Workout planner
`JEFit
`JeFit inc.
`Workout planner
`Workout coach
`Culleware
`Exercise book
`Fitness expert
`R4z0
`Exercise book
`Pilates
`Saulius
`Exercise book
`Yoga
`SusaSoftX
`
`Version
`1.37
`3.3.0
`2.0
`2.4.1.10
`1.4
`3.3.2
`1.8.5
`1.0
`1.0
`1.2.0
`1.1.2
`3.2.1208
`2.01
`1.5
`1.0
`1.4
`
`Downloads
`50.000–250.000
`>250.000
`>1.000.000
`>250.000
`50.000–250.000
`>250.000
`>500.000
`5000–10.000
`100–500
`50.000–250.000
`10.000–50.000
`>250.000
`10.000–50.000
`5000–10.000
`5000–10.000
`50.000–250.000
`
`Rev.
`6342
`28326
`11843
`5364
`529
`5999
`15963
`12
`5
`662
`232
`6342
`79
`32
`57
`293
`
`⋆
`
`4.5
`4.5
`4.5
`4.6
`4
`4.5
`4.5
`4.5
`4.5
`4
`4
`4.5
`3.5
`3
`3
`4
`
`running [18]. Beyond applications in sports, similar systems have been employed to assess professional skills, e.g. in surgery
`using a sensor-enhanced glove [19] or to assess metal filing skill [20], that have applications in training and evaluation.
`UbiFit Garden [21] brings activity recognition to the smartphone. For a certain amount of physical activity, flourishing
`flowers appear on the phone’s display as a motivational component. BALANCE [22] estimates the calorie expenditure in
`everyday life, contributing to long-term wellness management. Both smartphone solutions rely on additional sensors worn
`on the body.
`A multitude of commercial health devices and sensors, such as oximeters and heart rate monitors, formerly reserved for
`professional use, are now available and can be connected to smartphones. GPS watches, pedometers and heart rate monitors,
`allow recording and tracking of physical activity. For home use, hardware platforms like Nintendo Wii or Microsoft Kinect
`encourage users to physical activity, without focus on correct execution. Activity loggers like activPal1 or FitBit2 monitor
`health-related data and help create an activity profile. However, those solutions build upon dedicated systems or external
`sensor hardware. Since we want to motivate smartphone solutions working the real world without the need for additional
`hardware, the state of the art in health and fitness assistants on the smartphone is investigated subsequently.
`
`2.2. Comparative review of smartphone apps for health and fitness
`
`We evaluated the design space of current (beginning of 2011) popular health and fitness smartphone apps in a
`comparative review. For our review, we considered applications from the Health and Fitness category in the Android Market.
`A comparison with other app repositories (iTunes App Store and Nokia OVI Store) revealed that the offerings on different
`platforms are very similar. We chose Android for our review, being the platform with the highest coverage in 2011 and the
`fastest growing one. The heterogeneity of Android devices makes the platform also very suitable for fitness support, as the
`user can select a device adequately for personal needs (e.g. a small and light one on the run, and larger-screen devices for
`indoor usage).
`In order to reflect the highest quality and most satisfying apps available at investigation time, we used the Most Popular
`list in the Android Market and only considered apps with at least 3-star user ratings. We chose 16 apps as a representative
`sample for this review according the procedure described above. Except Workout Planner, which costs $1.99, all apps
`were available for free. Endomondo Sports Tracker, SportsTracking Sports Tracker, Adidas miCoach, RunKeeper Free, Cardio
`Trainer, AndAndo, JEFit, C25K, Daily Ab Workout Free, Body Fitness, Abs & Core, Workout Coach, Fitness Expert, Pilates, and
`Yoga (see Table 1). Some of them offer additional functionality in a pay version, but do not show a qualitative difference.
`
`2.2.1. Heuristic evaluation
`Quality assessments were created in a heuristic evaluation of the selected apps, performing a three-step analysis:
`
`1. The core task functionality was tested (task-focused walkthrough).
`2. Optional features were explored (explorative walkthrough).
`3. The descriptions by the app developers in the Android Market were compared to the actual functionality.
`
`1 http://www.paltech.plus.com/products.htm.
`2 http://www.fitbit.com/.
`
`Petitioner Apple Inc. – Ex. 1020, p. 3
`
`

`

`4
`
`M. Kranz et al. / Pervasive and Mobile Computing (
`
`)
`
`–
`
`Four heuristics were used to cover the most important aspects of mobile health and fitness support, which are explained
`below.
`App utility and usability for regular training. The usability for daily application in context of the supported activities was eval-
`uated. Factors observed in this area are the adequacy of interaction (size of controls, appropriate output, comprehensibility)
`and the customizability, i.e. whether the user can tailor the app to his needs. It was also assessed whether the app makes
`‘smart’ use of data (reuses once entered information, infers information from known data to minimize explicit input).
`Instructional quality of apps. It was examined how well the instructions serve to enable autonomous training guided by the
`mobile device. This includes the comprehensibility and extent of instructions, sufficient level of detail, etc.
`Sensor data usage. We evaluated to what extent sensor information is recorded and used by training and fitness applications.
`This includes sensor data recording for later review (e.g. GPS traces, acceleration data), inference of activity types, assessment
`of exercising skills, and incorporation of recent developments of Pervasive Computing, e.g. location determination to adapt
`training to indoor and outdoor environments, context-sensitivity and multi-user interaction.
`Motivation. It was examined how well an application is suited to generate and maintain long-term motivation and
`engagement, through e.g. diversity in the training experience, cooperative training (social incentive), and feedback provision
`on the training progress to maintain extrinsic motivation.
`
`2.2.2. Results and discussion
`We identified three categories of applications: GPS trackers, workout planners, and exercise books, each of which might
`include an option to connect, share or otherwise use the data in the context of social networks.
`GPS trackers. Apps in this category (see Table 1) annotate outdoor activities like running or cycling with location information.
`After the training, the GPS traces can be reviewed. Further information from the phone’s built-in sensors, such as
`accelerometer or magnetometer data, is usually not included. Some do, however, allow the connection to e.g. heart rate
`monitors and adapt training instructions to the heart rate (miCoach).
`Workout planners. These apps accompany goal-directed workout such as bodybuilding or weight training. They typically
`contain exercises organized by body parts or muscle groups and log exercise performances (quantitative, not qualitative).
`While the functionality and usability of apps in this category greatly varies, none offer exercising monitoring in terms of
`quality or individual performance.
`Exercise books. These apps provides a browsable compendium of exercises, with usually the least functionality, compared to
`the other genres, but the deepest background information on correct exercise performance and health-related issues.
`Summarizing our findings of the evaluation, the following points can be concluded.
`• The diversity of health and fitness apps is limited, despite a great number of apps in the respective category in the Android
`Market. Mobile fitness apps either focus on the recording part (which we called GPS loggers), or the instructional part
`(which we called exercise books). GPS loggers record location information so that a trace of the cardio activity becomes
`visible. A combination of exercise description and logging are workout planners, which often combine the weak parts
`of both sides: They just log that a certain exercise was performed, and instructions are not as detailed as in the exercise
`book category. The ideal would be combining the advantages from all three categories.
`• GPS loggers have the highest popularity (according to the number of downloads) in the Android Market. The reason
`probably lies in the high popularity of running, cycling and similar activities and the variety of supported activities by
`these apps. It can also be seen as an indicator in favor of a comprehensive approach of supporting fitness in different
`situations and locations. Body Fitness already picked up this idea by integrating both workout and cardio into one app,
`as well as supplements like a BMI (Body Mass Index) calculator and recipes. However, apps do not support yet pervasive
`training (at home, in the office, outdoors, ...) with activity recognition and contextually suited exercises.
`• Multi-user support is not yet integrated, but could be enabled by the creation of user profiles on the device could enable
`multi-user support also for logging applications and workout planners.
`• Few applications are compatible with sports hardware, such as heart rate monitors or wearable sensors. Neither
`do smartphones use the built-in sensors in current health and fitness applications, although almost all modern
`smartphones integrate accelerometers, magnetometers and gyroscopes. Device sensor usage could comfortably enable
`more functionality, without the need to buy, connect and synchronize hardware.
`• While some apps allow uploading training information to a portal for later review, none provide individualized,
`immediate and thereby motivating feedback directly on the phone on the exercise just performed.
`• The ‘advice’ provided by current systems such as e.g. the Adidas miCoach is on a more general level and requires educated
`interpretation and therefore is not really comparable to expert-like assessment of the individual exercises.
`• Current health and fitness apps only partially focus on long-term motivation so far. Social connections, e.g. to beat a
`friend’s results or to challenge a stranger, can be a motivational factor. The Facebook and Twitter integration offered by
`some apps is presumably rather motivated by promoting the app. Information on long-term usage is missing in any of
`the reviewed app’s descriptions in the Market. If an app would explicitly address motivation, this could be an enormous
`benefit for advertisement. We believe that adding individualized and personalized feedback could increase the long-term
`motivation of the user. We contribute to this high-level goal by facilitating this feedback on mobile devices.
`
`Petitioner Apple Inc. – Ex. 1020, p. 4
`
`

`

`M. Kranz et al. / Pervasive and Mobile Computing (
`
`)
`
`–
`
`5
`
`2.3. Lessons learned and recommendations
`
`Based on this review, we deduce some general guidelines for future health and fitness apps. The evaluation results show
`great potential for improvement in the reviewed disciplines usability, instruction quality and fostering motivation. Sensing
`and context information hereby both play an important role.
`Usability improvement. Unnecessary interaction with the device can be reduced e.g. by activity recognition. Research
`investigated activity detection with wearable sensors [4–6,23]. These techniques need to be tailored to smartphones, which
`are already equipped with a multitude of sensors. Evaluating acceleration and gyroscope data, could make the recognition
`of the performed activity possible, making a manual activity selection in the app obsolete. Also the activity-related calorie
`expenditure can thereupon be estimated. This results in a far more accurate determination of burnt calories after cardio
`training, as e.g. pauses or speed changes are taken into account.
`Instructional quality. The quality of exercise instructions can be improved through well-founded information and
`physiological correctness. Moreover, the combination with skill assessment and targeted feedback could be a large step
`towards self-determined, autonomous training. In physiotherapy or rehabilitation, the correct exercise performance is
`important for the healing process. The presence of a doctor or physiotherapist is not always possible, and permanent
`supervision not comfortable. Accurate exercise assessment based on sensor information and individualized feedback
`supports correct exercise accomplishment. Mobile phone feedback could thus be a valuable extension – but no replacement
`– to physiological care.
`Long-term motivation. We pointed out the importance of intrinsic motivation for upholding regular physical activity. The
`integration of feedback in sports and fitness applications adds to establish long-term motivation and engagement in various
`ways. First, a training assessment for singular exercises in the form of a score (‘you reached 85/100 points’) motivates beating
`this value and improving further to reach perfection. Second, a history of training results allows tracking improvement,
`acknowledging that regular training ‘pays off’. More sophisticated skill assessment, which we report on, could even enable
`targeted feedback. Apps could identify aspects of exercises with potential for improvement or indicate body parts that need
`particular training. Based on this analysis, the app could suggest suitable exercises addressing ‘sticking points’ and create a
`tailored training plan. Training assistance would become more efficient and help to reach goals faster.
`The integration of playful aspects and connection to social networks can further contribute to long-term motivation.
`Some applications already upload training results to Facebook, but real cooperative training needs to go further. Viewing
`friends’ high scores, and beating competitors make physical activity more fun than solitary training. These apps, though,
`miss a professional assessment of the training quality. We try to close this gap by the presented research.
`A step into this direction are apps such as Nike Training Club,3 an iPhone app (workout planner) combined with a social
`network, advertising with long-term motivation, but it does not incorporate automated activity recognition. Based on skill
`assessment, apps could also propose individualized training goals so that they are challenging, but not frustratingly hard.
`This also has an impact on social sports apps, as then relative comparisons between users would be possible. This, we do
`argue, will further support long-term motivation.
`Intelligent exercise assessment and monitoring is also relevant for elderly people, e.g. in an Ambient Assisted Living
`context. The smartphone application could be a reminder and a motivational factor to support physical activity, which is
`important for health risk reduction.
`
`3. GymSkill: automated assessment for balance board training
`
`With GymSkill [24,25], we present a smartphone application that addresses the shortcomings described above and
`introduces individualized exercise skill assessment fully based on integrated sensors. We describe the iterative design
`process, qualitative user feedback and quantitative validation of our assessment algorithm.
`
`3.1. Balance board training
`
`We chose balance board training as a representative discipline for physical exercising support, applicable to a wide range
`of young and old users, both for fitness training and rehabilitation [26,27]. With help of a sports medicine specialist, a set of
`20 exercises was conceived and fed into the GymSkill application.
`
`3.2. The GymSkill application
`
`GymSkill is implemented as an Android application and consists of an exercise database, sensor data recorder and the skill
`assessment presentation (see Fig. 3). During the performance, the smartphone (app running) is placed on top of the balance
`board (see Fig. 1) so that it can record all of its movements. The user begins her training by selecting an individual exercise or a
`complete training plan (see Fig. 4). Each exercise is explained step by step in text and pictures. During the actual performance,
`the phone records accelerometer and magnetometer data as a basis for the assessment which is presented to the user after
`completion of each exercise. GymSkill is freely available in Google Play as a research app under the name ‘GymSkill’.
`
`3 http://www.nike.com/nikewomen/features/faqs?locale=en_GB.
`
`Petitioner Apple Inc. – Ex. 1020, p. 5
`
`

`

`6
`
`M. Kranz et al. / Pervasive and Mobile Computing (
`
`)
`
`–
`
`Fig. 1. Left: GymSkill setup with smartphone placed on a balance board. Right: subject training with GymSkill.
`
`3.3. Case study
`
`A case study was conducted to collect initial sensor data for automatic exercise assessment, and to collect qualitative
`user feedback on satisfaction with the GymSkill application. We recruited six users (2 females, 4 males) aged from 25 to
`33 years (average age: 29) to perform sessions of 20 balance board exercises twice a day for five days, thus collecting 1200
`exercise records.
`
`3.3.1. Ground truth collection
`For all performed exercises performed in the study, we recorded accelerometer data as ground truth using a Nexus One
`smartphone that was fixed to the balance board using a rubber mat. The performances were recorded on video and their
`quality was assessed by an expert. A rating scheme of 11 individual aspects such as movement angles, regularity and tempo
`was used, adding up to 100 possible points.
`
`3.3.2. Qualitative feedback
`After the training sessions, participants were interrogated on the GymSkill application. Since the prototype used for
`this study was not yet able to provide real-time feedback, the questions focused on the handling of the application and
`its potential to motivate for regular training. The need for online feedback that was requested by ten participants at the
`end of this study, has been implemented in the second iteration presented later. A summary of the results can be seen in
`Fig. 2. Subjects stated that GymSkill could help to reach a training goal faster with an average of 4.2 on a 5-step Likert scale
`(1 = fully disagree, 5 = fully agree), SD = 1.3. GymSkill’s potential to maintain long-term motivation was confirmed with
`an average of 3.7 (SD = 1.0). Asked for the most attractive potential features of a personal fitness trainer like GymSkill,
`individual exercise feedback was named with 5.0 (SD = 0.0), followed by individual exercise suggestions (4.8, SD = 0.4).
`Subjects stated that they would use the application regularly with averagely 3.6 (SD = 1.0). The handling of the system
`(placing the phone on the board to record data) was evaluated as easy (average agreement: 4.0, SD = 1.1); placing the
`phone on the board to record data was apparently not perceived as problematic.
`
`3.4. First iteration: Server-based automatic exercise assessment
`
`In the first iteration, GymSkill was a two-part system consisting of the app itself and a server analysis component (see
`Fig. 3, left diagram).
`The logged information is sent to a server, where the automatic analysis is performed in terms of a complex retrospective
`assessment of the exercise correctness, which serves as an indicator of the user’s skill. The calculated skill level sent back
`to the mobile phone, indicated as visual feedback in different steps between ‘thumbs up’ and ‘thumbs down’. Additionally,
`more sophisticated graph visualizations are available, which allow the review of the exercise over time and help the user to
`identify specific problems.
`During training, simple feedback like the remaining number of repetitions or excessive displacement of the board is is
`signaled by a warning sound and visually.
`
`3.4.1. Quality measures
`In order to assess the quality of exercises, it is important to estimate global quality measures that cover the important
`aspects of the performed motion. Advised by an expert clinician, the following attributes were defined.
`
`Petitioner Apple Inc. – Ex. 1020, p. 6
`
`

`

`M. Kranz et al. / Pervasive and Mobile Computing (
`
`)
`
`–
`
`7
`
`Fig. 2. Left: after 5 days of training with a GymSkill prototype, users believe the app could motivate and help to reach training goals faster. Right: individual
`feedback and suggestions of exercises were particularly attractive features. Answers given on a Likert scale (1 = fully disagree, 5 = fully agree). The box
`indicates the interquartile range (middle 50%), the dot the mean value, and the bars minima and maxima.
`
`(a) Iteration 1: the smartphone records sensor data during exercising,
`which are processed by a server after the training to generate skill
`assessment.
`
`(b) Iteration 2: data is processed on the phone for
`real-time feedback as well as sophisticated feedback
`addressing individual aspects after the execution.
`
`Fig. 3.
`
`Iterations of the GymSkill application.
`
`Smoothness and continuity of movement. For continuous exercises, as they are typical for gym-based training, it is important
`to maintain smooth motion. In order to remain relatively independent of the particular exercise and to avoid the excessive
`use of prior knowledge, a novel local assessment approach has been developed (next section).
`Global motion quality. Each exercise requires the user to perform particular motion sequences. The assessment on how well
`these motions were performed is crucial for the assessment of the quality of the performed task.
`Usage of board’s degrees of freedom. If a task requires the user to fully displace the board along at least one degree of freedom,
`the fraction to which he uses this opportunity while avoiding extreme postures (e.g., touching the ground) provides a
`valuable measure for exercise performance.
`The goal of the automated assessment is to estimate measures for the aforementioned aspects and to combine them
`into a single performance criterion or metric. Aiming at transferability of the method the amount of parameters and prior
`knowledge used us limited as much as possible.
`Before the actual analysis the recorded orientation-values (azimuth, pitch, roll) are normalized to a common value-range
`with zero-mean. Devi