as) United States
`a2) Patent Application Publication co) Pub. No.: US 2006/0010699 A1
`(43) Pub. Date: Jan. 19, 2006
`
`Tamura
`
`US 20060010699A1
`
`(54)
`
`(75)
`
`MOBILE TERMINAL APPARATUS
`
`Publication Classification
`
`Inventor: Yasuhiro Tamura, Tokyo (JP)
`
`Correspondence Address:
`Ralph A. Dowell of DOWELL & DOWELL P.C.
`2111 Eisenhower Ave.
`Suite 406
`
`Alexandria, VA 22314 (US)
`
`(73)
`
`Assignee: C&N Inc.
`
`(21)
`
`Appl. No.:
`
`11/067,210
`
`(22)
`
`Filed:
`
`Feb. 28, 2005
`
`Related U.S. Application Data
`
`(63)
`
`Continuation of application No. PCT/JP04/10140,
`filed on Jul. 15, 2004.
`
`(51)
`
`Int. Cl.
`(2006.01)
`GOIC 17/02
`(52) US. Ch.
`ceecsscssssssssssssnsnetnnistsetnsvesnsn 33/355 R
`
`(57)
`
`ABSTRACT
`
`Amobile terminal apparatus 1 uses an acceleration sensor as
`a tilt angle sensor 24,and realizes various applications based
`on this detection result. A processing unit 10 counts the
`number of steps of a human based on acceleration compo-
`nents of low frequency detected bythe tilt angle sensorthat
`detects the acceleration components. Atthat time, the accel-
`eration componentof an axis among three axes which most
`approximates a gravity axis 1s mainly used. A viewpoint
`relative to a three-dimensional object or a three-dimensional
`space displayed on a display unit 36 is moved. Byeffectively
`utilizing the detection result of the tilt angle sensor,
`it is
`possible to realize a mobile terminal apparatus 1 featuring
`greater usefulness.
`
`
`
`
`
`TILT
`MAGNETIC
`
`ANGLE
`SENSOR
`
`
`SENSOR
`
`PROCESSING
`UNIT
`
`GPS
`INFORMATION
`ACQUIRING
`UNIT
`
`
`
`
`
`DISPLAY
`UNIT
`
`Page | of 12
`
`SAMSUNG EXHIBIT 1005
`
`Page 1 of 12
`
`SAMSUNG EXHIBIT 1005
`
`

`

`Patent Application Publication Jan. 19,2006 Sheet 1 of 5
`
`US 2006/0010699 Al
`
`FIG.1
`
`BAROMET-
`
`SENSOR
`
`SENSOR
`
`30
`
`
`
`
`
`
`
`
`TILT
`
`
`
`SENSOR
`
`
`
`
`
`IMAGE
`
`PICKUP
`UNIT
`
`
`10
`
`34
`GPS
`ce=:
` PROCESSING
`
`INFORMATION
`ACQUIRING
`UNIT
`
`
`
`
` 32
`
`COMMUN | —
`CATION
`UNIT
`
`| DISPLAY
`UNIT
`
`Page 2 of 12
`
`Page 2 of 12
`
`

`

`Patent Application Publication Jan. 19, 2006 Sheet 2 of 5
`
`US 2006/0010699 Al
`
`FIG.2
`
` DETECT
`
`TERRESTRIAL
`
`MAGNETISM
`
`
`
`
`
`COORD I NATE
`TRANSFORMATION
`
`
`
`
`
`CALCULATE
`AZIMUTH
`
`
`Page 3 of 12
`
`Page 3 of 12
`
`

`

`Patent Application Publication Jan. 19, 2006 Sheet 3 of 5
`
`US 2006/0010699 Al
`
`FIG.3
`
`40
`
`Page 4 of 12
`
`Page 4 of 12
`
`

`

`Patent Application Publication Jan. 19, 2006 Sheet 4 of 5
`
`US 2006/0010699 Al
`
`FIG.4
`
`Page 5 of 12
`
`Page 5 of 12
`
`

`

`Patent Application Publication Jan. 19, 2006 Sheet 5 of 5
`
`US 2006/0010699 Al
`
`FIG.5A -
`
`FIG.5B
`
`FIG.5C
`
`Cy
`
`
`
`Page 6 of 12
`
`Page 6 of 12
`
`

`

`US 2006/0010699 Al
`
`Jan. 19, 2006
`
`MOBILE TERMINAL APPARATUS
`
`FIELD
`
`[0001] The present invention relates to mobile terminal
`apparatuses such as a mobile telephone and it particularly
`relates to a mobile terminal apparatus equipped with a
`function to determine azimuth.
`
`BACKGROUND
`
`[0002] Developing mobile terminal apparatuses equipped
`with azimuth measuring function has been conducted in
`recent years. A magnetic sensor is utilized for the measure-
`ment of an azimuth. These mobile terminal apparatuses
`display a present position and the like on a map shown on
`the screen using an azimuth measured according to posi-
`tional information from GPS (Global Positioning System).
`
`[0003] The magnetic sensor detects not only natural mag-
`netic fields but also dynamic magnetic fields generated by
`the interior of the mobile terminal apparatus as well as by
`nearby electrical equipmentor the like. Hence, in order to
`extract natural magnetic field component only, it is neces-
`sary to delete the dynamic magnetic components from the
`detected magnetic components. When a two-axis magnetic
`sensor is used, the magnetic field strength cannot be deter-
`mined,so thatit is difficult to remove the dynamic magnetic
`componentsefficiently.
`
`[0004] Consideration is given to the introduction of a
`three-axis magnetic sensor in order to determine the mag-
`netic field strength. Moreover,
`to determine an azimuth
`accurately, it is necessary to correct the terrestrial magnetism
`vector component detected by the three-axis magnetic sen-
`sor with the inclination of the mobile terminal apparatus
`taken into consideration. The inclination of the mobile
`terminal apparatus is measured bya tilt angle sensor.
`
`SUMMARY OF THE INVENTION
`
`[0005] Under these circumstances, it is preferable that a
`tilt angle sensor be provided in a mobile terminal apparatus
`equipped with azimuth measuring function. The inventor of
`the present invention has developed a variety of applications
`by focusing his attention on the effective utilization of this
`tilt angle sensor. The object of the present
`invention is
`therefore to provide a variety of possibilities of a mobile
`terminal apparatus equipped with azimuth measuring func-
`tion and realize a mobile terminal apparatus featuring
`greater usefulness.
`
`[0006] A mode of the present invention is a mobile ter-
`minal apparatus. This apparatus comprises: a magnetic sen-
`sor which detects three-axis components of a terrestrial
`magnetism vector; a tilt angle sensor which detects accel-
`eration components in three-axis directions; a processing
`unit which determines azimuth by performing coordinate
`transformation on the terrestrial magnetism components
`detected by the magnetic sensor, based on a detection result
`of the tilt angle sensor, wherein the processing unit detects
`acceleration components of a frequency in a predetermined
`range, based on a detection result along an axis among the
`axes of the tilt angle sensor which most approximates a
`gravity axis, and counts the numberof user’s steps. Accord-
`ing to this mode, the provision of a highly accurate function
`of a pedometer can result in realizing a mobile terminal
`apparatus featuring greater usability.
`
`[0007] The processing unit may determine user’s ambu-
`lation trajectory based on the azimuth and the numberof
`steps or may omit a componentthat has converged within a
`predetermined period of time, among the detected accelera-
`tion components, from the count of the numberof steps. By
`omitting vibrations or shocks other than walking motion
`from the count of the number of steps, the pedometer is
`given a function having further raised accuracy.
`
`[0008] Another mode of the present invention is also a
`mobile terminal apparatus. This apparatus comprises: a
`magnetic sensor which detects three-axis components of a
`terrestrial magnetism vector; a tilt angle sensor which
`detects acceleration components in three-axis directions; a
`display unit which displays an image from a predetermined
`viewpoint relative to a three-dimensional object or a three-
`dimensional space; and a processing unit which determines
`azimuth by performing coordinate transformation on the
`terrestrial magnetism components detected by the magnetic
`sensor, based on a detection result of the tilt angle sensor,
`wherein the processing unit moves the viewpointrelative to
`the three-dimensional object or the three-dimensional space,
`based on a detection result of the magnetic sensor and the tilt
`angle sensor. The processing unit may obtain yaw angle,
`pitch angle androll angle based on the detection result of the
`magnetic sensor and the tilt angle sensor, so as to be
`corresponded to a three-dimensional shift of the viewpoint.
`According to this mode, a mobile terminal apparatus can be
`realized in which the three-dimensional object or the three-
`dimensional space can be observed from all directions.
`
`[0009] Still another mode of the present invention is also
`a mobile terminal apparatus. This apparatus comprises: a
`magnetic sensor which detects three-axis components of a
`terrestrial magnetism vector; a tilt angle sensor which
`detects acceleration components in three-axis directions; a
`display unit which displays map; and a processing unit
`which determines azimuth by performing coordinate trans-
`formation on the terrestrial magnetism components detected
`by the magnetic sensor, based on a detection result of the tilt
`angle sensor, wherein, based on a detection result of the
`magnetic sensorandthe tilt angle sensor, the processing unit
`rotates the map in such a manneras to align azimuth of the
`map with actual azimuth. According to this mode, a display
`is possible in which the mapis related to the actual azimuth.
`
`[0010] Still another mode of the present invention is also
`a mobile terminal apparatus. This apparatus comprises: a
`magnetic sensor which detects three-axis components of a
`terrestrial magnetism vector; a tilt angle sensor which
`detects acceleration components in three-axis directions; an
`image pickup unit which acquires moving images; and a
`processing unit which determines azimuth by performing
`coordinate transformation on the terrestrial magnetism com-
`ponents detected by the magnetic sensor, based on a detec-
`tion result of the tilt angle sensor, wherein, based on a
`detection result of the magnetic sensor and the tilt angle
`sensor,
`the processing unit corrects camera-shake in the
`moving images. Whenthe processing unit detects a camera-
`shake based on the detection result of the magnetic sensor
`and the tilt angle sensor, correction may be made in such a
`manner that a feature point of a frame at the detection of
`camera-shake is aligned to that of a frame preceding the
`frame. According to this mode, a mobile terminal apparatus
`equipped with a camera having a function of correcting
`camera-shake can berealized.
`
`Page 7 of 12
`
`Page 7 of 12
`
`

`

`US 2006/0010699 Al
`
`Jan. 19, 2006
`
`[0011] The apparatus in the above-described modes may
`further comprise a barometric sensor which detects the
`atmospheric pressure, wherein the processing unit may
`predict
`the weather based on a detection result of the
`barometric pressure. A mobile terminal which can predict
`the weather can be realized.
`
`It is to be noted that any arbitrary combination of
`[0012]
`the above-described elements and expressionsofthe present
`invention changed among a method, a system, a recording
`medium, computer program and so forthare alsoeffective as
`the modeof the present invention.
`
`netic sensor 22. The processing unit 10 may be used in such
`a mannerthat the positional information and the azimuthal
`information complementeach other. The processing unit 10
`can also determine a present position and azimuth based on
`the detection results only from the detecting unit 20 without
`making use of the positional information from GPS satel-
`lites. The display unit 36, which has a display, outputs
`information processed at the processing unit 10 according to
`an application. It is to be noted that a speaker, not shown,
`may be provided to produce audio outputs of various types
`of information for the user.
`
`[0020] The magnetic sensor 22 detects the three-axis com-
`ponentsofterrestrial magnetism vector, and may be any type
`including flux gate type sensors and those using a Hall
`element or magnetoresistive element. The X axis, Y axis and
`Z axis are placed orthogonally to one another. The azimuth
`angle, or the yaw angle, which is the angle of rotation from
`magnetic north, is detected based on the X-axis and Y-axis
`FIG.2 is a figure showing a flow of calculating an
`[0014]
`terrestrial magnetism vectors. The terrestrial magnetism
`azimuth based on detection results from a magnetic sensor
`vectors, however, are approximately horizontal near the
`andatilt angle sensor.
`equator, but are inclined in other locations. Also, the posture
`of the mobile terminal apparatus 1 is not always horizontal.
`Hence, such terrestrial magnetism vectors have to be cor-
`rected, using the pitch angle and roll angle, which are the
`angles relative to the direction of gravity. The pitch angle
`and the roll angle are detected by the tilt angle sensor 24.
`The processing unit 10 corrects the X-axis and Y-axis
`detection results, using the above-mentioned pitch angle and
`roll angle, in such a mannerthat the Z-axis detection results
`are always equivalent
`to the condition where they are
`maintained at
`the detection results when the Z axis is
`horizontal. In this manner, even whenthereis a great change
`in posture, the accuracy of azimuth angle can be retained by
`employing a three-axis magnetic sensor and using the Z-axis
`detection results for correction.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0013] FIG. 1 is a figure showing a structure of a mobile
`terminal apparatus according to an embodiment of the
`present invention.
`
`FIG.3 is a figure showing an example in which a
`[0015]
`three-dimensional object is displayed on a display unit.
`
`FIG.4 is a figure showing an example in which a
`[0016]
`map is displayed on a display unit.
`
`[0017] FIGS. 5 are figures showing how camera-shake is
`corrected by a processing unit. (a) shows a frame immedi-
`ately preceding a frameat the detection of camera-shake; (b)
`shows a frame at the detection of camera-shake; and (c)
`shows how frames were synthesized.
`
`DETAILED DESCRIPTION
`
`[0018] FIG. 1 shows a structure of a mobile terminal
`apparatus 1 according to an embodiment of the present
`invention. This mobile terminal apparatus 1 is a portable
`small electronic device such as cellular phone, PHS (Per-
`sonal Handyphone System) and PDA(personal data assis-
`tant). FIG. 1 showsa structure for realizing azimuth mea-
`surement and applications to be described later, but it is
`understood by those skilled in the art
`that
`the mobile
`terminal apparatus 1 may haveotherstructures according to
`its uses.
`
`[0019] The mobile terminal apparatus 1 is comprised of a
`processing unit 10, a detecting unit 20, an image-pickup unit
`30, a communication unit 32, a GPS information acquiring
`unit 34 and a display umit 36. The detecting unit 20 has a
`magnetic sensor 22, a tilt angle sensor 24, a barometric
`sensor 26 and a temperature sensor 28, and has a function of
`detecting position, azimuth, bearing, altitude and so forth.
`The image-pickup unit 30, which is equipped with a pho-
`toelectric conversion device, such as a CCD, acquires an
`image and sends it to the processing unit 10. The commu-
`nication unit 32 has a function of communicating with an
`external server via a wireless circuit. This communication
`
`may also be carried out via a wire circuit. The GPS infor-
`mation acquiring unit 34 receives positional information
`from GPSsatellites. The processing unit 10 calculates a
`present position, namely, the latitude and longitude, based
`on this positional information. It is to be noted that an
`accurate present position and azimuth can be determined by
`an arrangement that makesit possible to correct the posi-
`tional information by azimuthal information from the mag-
`
`Page 8 of 12
`
`Thetilt angle sensor 24, which is an acceleration
`[0021]
`sensor for detecting acceleration components in three axis
`directions, comes in the resistance value change type, the
`capacity change type, the piezoelectric change type and so
`forth. For the tilt angle sensor 24, too, the X axis and the Y
`axis are placed orthogonally to each other in a horizontal
`plane, and the Z axisis placed in the direction of gravity. As
`the posture of the mobile terminal apparatus 1 inclines, the
`gravitational acceleration changes, and the pitch angle and
`the roll angle are detected by detecting this gravitational
`acceleration. When the mobile terminal apparatus 1 is in the
`state of being stationary, the posture of the mobile terminal
`apparatus 1 can be detected accurately using the two axes
`only. In a case when the user who possesses the mobile
`terminal apparatus 1 is walking or riding a vehicle such as
`a car or a bicycle, a kinetic acceleration component will be
`added to have effect on the tilt angle sensor 24, so that the
`posture cannot be detected accurately. By the use of three
`axes, the gravitational acceleration and the kinetic accelera-
`tion can be separated from each other, thereby enabling an
`accurate detection of the posture. The pitch angle and roll
`angle can be calculated accurately by obtaining an assumed
`angle by integrating the output values of the respective axes
`and then performing a predetermined calculation by com-
`paring the assumed angle with the acceleration component.
`
`[0022] The barometric sensor 26 detects the atmospheric
`pressure, and the temperature sensor 28 detects the tempera-
`ture. The temperature detected is used to correct the devia-
`
`Page 8 of 12
`
`

`

`US 2006/0010699 Al
`
`Jan. 19, 2006
`
`tions of the outputs of the magnetic sensor 22, the tilt angle
`sensor 24 and the barometric sensor 26 due to temperature
`drift.
`
`[0023] FIG. 2 shows a flow of calculating an azimuth
`based on detection results from a magnetic sensor 22 and a
`tilt angle sensor 24. First, the tilt angle of the magnetic
`sensor 22 is calculated based on the acceleration compo-
`nents in the three-axis directions detected by the tilt angle
`sensor 24 (S10). Here, the pitch angle and the roll angle are
`components of the terrestrial magnetism vector (S12). The
`processing unit 10 performs a coordinate transformation of
`the components of the terrestrial magnetism vector by the
`use of the pitch angle and the roll angle (S14) and thereby
`determines an accurate azimuth (S16). At
`this time, a
`temperature compensation may becarried out, using the
`detection results from the temperature sensor 28. This way,
`the detection results from the tilt angle sensor 24 are used in
`the calculation of azimuth; however, the present invention,
`by further utilizing the detection results effectively, realizes
`applications as described hereinbelow.
`
`<Application 1>
`
`In Application 1, a mobile terminal apparatus 1 is
`[0024]
`provided with the function of a pedometer. A tilt angle
`sensor 24 detects acceleration components in the three-axis
`directions in accordance with the movement of the mobile
`terminal apparatus 1. Thus, while a user in possession of a
`mobile terminal apparatus 1 is walking,the tilt angle sensor
`24 detects not only the acceleration components correspond-
`ing to the tilt angle of the mobile terminal apparatus 1 but
`also low-frequency acceleration components in response to
`the movementof the user.
`
`[0025] The processing unit 10 performs a frequency analy-
`sis on the detection results from the tilt angle sensor 24 and
`counts the numberof user’s steps based on the acceleration
`counts the numberof user’s steps based on the acceleration
`components of a frequency within a predetermined range.
`This predetermined rangeis set beforehand in such a manner
`as to cover the walking speedsof the user. An optimum value
`may be determined through simulation or actual measure-
`ment. Since walking motion is such as to add an acceleration
`component mainly in the direction of gravity, detection
`results along an axis within the tilt angle sensor 24 which
`most approximates the axis of gravity are used. Also, the
`detection results along an axis which most approximates the
`axis of gravity may be corrected based on the detection
`results along the other two axes. A decision on which axis
`most approximates the axis of gravity can be made suchthat,
`in a Stationary state, an axis showingthe largest gravitational
`acceleration componentis selected and, in a dynamicstate,
`it can be calculated based on changesin the resistance values
`of the respective axes and the calculated values of the pitch
`angle and the roll angle.
`
`Here, an optimum valuefor the predetermined length of time
`can be obtained by simulation and actual measurement of
`the vibration from walking motion of grownupsin general,
`shocks occurring when the mobile terminal apparatus 1 is
`dropped, and the like. The processing unit 10 can have the
`calculated numberof steps displayed on the display unit 36.
`It may also have consumed calories corresponding to the
`numberof steps displayed.
`
`[0027] The processing unit 10 can have a present position
`shown on a map displayed on the display unit 36, based on
`map data stored on ROM (not shown) and GPS information
`obtained from a GPSinformation acquiring unit 34. And the
`processing unit 10 can have the ambulation trajectory of the
`user displayed. The processing unit 10 can obtain ambula-
`tion trajectories from GPS information, but where the radio
`waves do notreach, it cannot obtain ambulation trajectories
`from GPS information. Also, in determining an ambulation
`trajectory by the use of GPS information,
`the walking
`distance from the starting point to the measuring point is
`calculated; however, an accurate azimuth cannot be calcu-
`lated unless a certain amount of distance is traveled in a
`
`certain length of time. Moreover, an azimuth, which is based
`on GPSinformation,is calculated after passing the measur-
`ing point, and therefore there is a possibility that it differs
`from the current value of azimuth.
`
`[0028] Thus, the processing unit 10 determines an ambu-
`lation trajectory based on the azimuth detected by the
`magnetic sensor 22 and the numberof steps and the stride
`detected bythe tilt angle sensor 24. Since thestride is nearly
`constant during a walk, the distance walked can be calcu-
`lated by multiplying the stride by the numberof steps. The
`stride to be used may bea set value or one inputted by the
`user. Alternatively, a stride obtained by calculating after first
`acquiring GPS information between two points may be
`registered.
`
`[0029] The processing unit 10 makes a choice for use as
`appropriate between calculated values based on GPS infor-
`mation and calculated values based on the magnetic sensor
`22 andthetilt angle sensor 24. For example, in places where
`radio waves from GPSsatellites cannot be received, control
`may be made so asto use the calculated valuesofthe latter.
`Furthermore, both the calculated values may be used to
`complementeachother, for instance, by using their averages
`or by comparing the two calculated values when there has
`occurred an abrupt change.
`
`<Application 2>
`
`In Application 2, a three-dimensional object or an
`[0030]
`image in a three-dimensional space displayed on a display
`unit 36 can be switched to an image, such as said object, seen
`from a different viewpoint by changing the orientation or
`posture of a mobile terminal apparatus 1. When the user
`changes the orientation or posture of the mobile terminal
`the
`[0026] Moreover, in counting the number of steps,
`apparatus 1, a magnetic sensor 22 andatilt angle sensor 24
`accuracy thereof can be further raised by making a distinc-
`detect the changed components thereof. A processing unit 10
`tion between the vibration due to walking motion and the
`calculates the yaw angle, pitch angle and roll angle based on
`other shocks. The vibration due to walking motion, which is
`the detection results.
`a regular motion, does not converge so long as the user is
`walking. In contrast thereto, simple shocks converge within
`a predetermined length of time. Accordingly, vibration com-
`ponents beyond this predetermined length of time are
`counted as walking, and the shocks that converge within the
`predetermined length of time are not counted as walking.
`
`FIG.3 is a figure showing an example in which a
`[0031]
`three-dimensional object is displayed on a display umit. The
`user can observe the three-dimensional object 40 displayed
`on the screen from all directions. In other words, it suffices
`if the viewpoint relative to the three-dimensional object 40
`
`Page 9 of 12
`
`Page 9 of 12
`
`

`

`US 2006/0010699 Al
`
`Jan. 19, 2006
`
`can be moved freely in the up-and-down direction, right-
`and-left direction and depth direction. The processing unit
`10 has the shift distances in these three directions corre-
`spond to the changes in the yaw angle, pitch angle androll
`angle which are calculated based on the detection results
`from the magnetic sensor 22 and the tilt angle sensor 24.
`These three correspondencerelations and the shift distances
`of the viewpoint in response to the respective angle changes
`can be set freely by a designer. Also, it may be so arranged
`that
`the user can change these settings arbitrarily. For
`example, when the change in yaw angle is corresponded to
`the shift in the depth direction, enlarged or reduced display
`can be realized as the user rotates the mobile terminal
`
`apparatus 1 while maintaining its posture. Moreover, it is
`possible to move to the back side of the three-dimensional
`object 40 by shifting the viewpoint. In this manner,
`the
`viewpoint can be changed freely without key operation if the
`user changes the orientation and posture of the mobile
`terminal apparatus 1.
`
`<Application 3>
`
`In Application 3, displays are made in a manner
`[0032]
`such that the azimuth on a map displayed on a display unit
`36 is associated with the actual azimuth. FIG.4 is a figure
`showing an example in which a map is displayed on the
`display unit 36. A processing unit 10 causes a map to be
`displayed on the display unit 36 based on mapdata stored in
`ROM(not shown)or the like. At the same time, it calculates
`the azimuth by correcting the detection results from a
`magnetic sensor 22 according to the detection results from
`a tilt angle sensor 24. The map displayed is rotated in such
`a mannerthat agreementis reached between the azimuth on
`the map displayed and the actual azimuth. Hence, the map
`displayed on the display unit 36 rotates as the user changes
`the direction he/she is facing.
`
`[0033] The processing unit 10 causes the north to be
`always displayedat the top of the screen withoutrotating the
`mapitself displayed on the display unit 36, but may have the
`user recognize the actual north by simultaneously providing
`a display of an electronic compass 53 or the like. This way,
`the user can gain useful information by a display made in a
`manner such the azimuth on the mapis associated with the
`actual azimuth.
`
`In the same wayas in Application 2, the viewpoint
`[0034]
`for the map image displayed on the display unit 36 can be
`switched by changing the orientation and/or posture of the
`mobile terminal apparatus 1. As the user changesthe orien-
`tation and/or posture of the mobile terminal apparatus 1, the
`magnetic sensor 22 and the tilt angle sensor 24 detect the
`changing components. The processing unit 10 calculates the
`yaw angle, pitch angle androll angle based on the detection
`results. The designer relates the changes in the pitch angle,
`roll angle and yaw angle to the shift distances in the
`up-and-down direction, right-and-left direction and height
`direction of a viewpoint 50 shown in FIG.4. For example,
`whenthe user movesthe mobile terminal apparatus 1 in such
`a manner as to change the yaw angle, the height from the
`map plane changes, so that an enlarged or reduced display
`can be produced. Likewise, when the user moves the mobile
`terminal apparatus 1 in such a manneras to change the pitch
`angle and roll angle, the map scrolls up or downorright or
`left. It is to be noted that, similar to Application 2,
`the
`designer can freely set the three correspondence relations
`
`and the shift distances of the viewpoint relative to the
`respective angle changes. Moreover, it may be so arranged
`that the user can change these settings arbitrarily.
`
`In Applications 2 and 3, the processing may be
`[0035]
`such that
`the viewpoint
`is shifted effectively when the
`angular velocity of the yaw angle, pitch angle and roll angle
`is at or above a predetermined threshold value and the
`viewpoint is not shifted when they are below the threshold
`value. An optimum value for a predetermined threshold
`value may be determined through simulation or actual
`measurement.
`In this case, when the user changes the
`orientation and/or posture of the mobile terminal apparatus
`1 at a speed below a predetermined speed,
`the image
`displayed on the display unit 36 will keep a stationary state,
`and said image will, for instance, scroll when he/she doesit
`at a speed at or above the predetermined speed.
`
`<Application 4>
`
`In Application 4, correction is made for camera-
`[0036]
`shake that occurs during the pickup of moving images by an
`image-pickup unit 30. If the camera-shake is caused by a
`user during the image taking of moving images by the
`image-pickup unit 30 of a mobile terminal apparatus 1, a
`magnetic sensor 22 and a tilt angle sensor 24 detect the
`changing components due to the camera-shake. The pro-
`cessing unit 10 checks on the changing components and
`determines whether the changes in the orientation and/or
`posture of the mobile terminal apparatus 1 are due to the
`camera-shake or not. For instance, it is possible that the
`processing is carried out in a mannersuch that vibrations or
`the like surpassing a predetermined threshold value are not
`regarded as camera-shake but slight vibrations below the
`predetermined threshold value is regarded as camera-shake.
`Here, an optimum value for the predetermined threshold
`value may be determined through simulation or actual
`measurement.
`
`[0037] FIGS. 5 are figures showing how camera-shakeis
`corrected by a processing unit 10.
`(@) shows a frame
`immediately preceding a frame at the detection of camera-
`shake; (b) shows a frame at the detection of camera-shake;
`and (c) shows how frames were synthesized. Upon detecting
`a camera-shake,
`the processing unit 10 extracts feature
`points of a frame immediately preceding the frame at the
`detection of the camera-shake. Extraction of feature points
`may be madebyutilizing the luminance components of each
`frame. In FIG. 5(a), the neck portion of a person is extracted
`as a feature point 60. Then, the feature point 60 is searched
`for in the frameat the detection of camera-shake. Preferably,
`the search for the feature point 60 is made within a range of
`about 10 percent up and down andright andleft of the pixel
`position of the feature point 60 in the immediately preceding
`frame. Whenextracting the feature point 60 in the frameat
`the detection of camera-shake, the processing unit 10 syn-
`thesizes the two frames by superposing the feature point 60
`in the frame at the detection of camera-shake on the feature
`
`point 60 in the immediately preceding frame. A state of this
`is shownin FIG.5(c). In FIG.6, the feature point 60 in the
`frameat the detection of camera-shakeisoff to the right, and
`therefore the synthesis of the two framesresults in a creation
`of a surplus portion at
`the left end of the frame at the
`detection of camera-shake. The processing unit 10 cuts off
`the surplus portion outside the synthesized range and then
`takes this as a frame at the detection of camera-shake.
`
`Page 10 of 12
`
`Page 10 of 12
`
`

`

`US 2006/0010699 Al
`
`Jan. 19, 2006
`
`In this manner, the camera-shakethat occurs during
`[0038]
`an image-pickup by a mobile terminal apparatus 1, such as
`a mobile phone with a built-in camera, can be easily
`a processing unit which is connected to the magnetic
`corrected by image processing effected by an application
`sensor and thetilt angle sensor, and which determines
`without the use of any complex mechanism. In doing so, the
`azimuth by performing coordinate transformation on
`correction may be made only when the occurrence of
`the terrestrial magnetism components detected by said
`camera-shake is recognized in the light of the detection
`magnetic sensor, based on a detection result of said tilt
`results from a magnetic sensor 22 andatilt angle sensor 24,
`angle sensor,
`so that the increase in the amount of computation can also
`be suppressed to the minimum.
`
`a tilt angle sensor which detects acceleration components
`in three-axis directions;
`
`<Application 5>
`
`In Application 5, the atmospheric pressure, altitude
`[0039]
`and weather are measured and displayed by the use of a
`barometric sensor 26. The altitude is calculated using the
`relationship of “the higher the altitude, the lower the atmo-
`spheric pressure”. Since the measured value of atmospheric
`pressure varies with the weather, it is preferable that the
`relationship between the pressures at the earth’s surface,
`namely, the absolute atmospheric pressures, and the rises
`thereof at higher positions above the earth’s surface, namely,
`the relative atmospheric pressures, be recorded in advance as
`a table in ROM (not shown)orthelike. It is to be noted that
`the relationship between the absolute atmospheric pressures
`and the relative atmospheric pressures may also be stored in
`a recording portion in the form of a computing equation. The
`processing unit 10 specifies the altitude by calculating the
`absolute atmospheric pressure and the relative atmospheric
`pressure based on the output value of the barometric sensor
`26.
`
`[0040] The processing unit 10 measures changes in atmo-
`spheric pressure by obtaining measurement data from the
`barometric sensor 26 at predetermined timeintervals. From
`these changes in atmospheric pressure,
`it
`is possible to
`predict the weather ahead as it is improving if the atmo-
`spheric pressure is rising, or as it is deteriorating if the
`atmospheric pressure is dropping.
`
`[0041] The present invention has been described based on
`embodiments. The embodiments are only exemplary and it
`is understood by those skilled in the art that there may exist
`various modifications to the combination of such