The base framework of our method is spacetime optimization, adjusting an original motion to fit user-specified constraints, including kinematic and force constraints. Nevertheless, the high complexity of human musculoskeletal model makes the optimization process time-consuming or difficult to converge to an acceptable solution. To overcome this problem, we observe that every human motion has considerable coherence. By the articulation of human skeleton, there is spatial relationship between neighbor joints; by the gravity and muscle force, there is temporal relationship between several frames. Therefore, we propose exploiting both spatial and temporal coherence to keep only the most effective dimensions.
Principle components analysis (PCA) is a space transformation technique by projecting multidimensional data sets to a lower dimensional space while retaining most significant features. When applying PCA on the motion data, it can hold the major moving information of the hierarchy. In
3
order to maximize the utilization of spatial coherence so that the moving information can be described more accurately in the low dimensional space, we divide each limb and torso into different segments and group the most correlated joints. As a result, we can use fewer DOFs to control the original motion properly in the optimization process. Temporal coherence makes the trajectories of motion vary continuously. We use splines to approximate the smooth of motion such that we only need to optimize key frames at regular intervals. Hereby, numbers of DOFs are further decreased in the phase of optimization.
Our editing system provides an interface for users to modify original motions by specifying high-level constraints, including kinematic constraints and force constraints. Kinematic constraints include end effectors positions, footsteps positions, etc. A force constraint is an external force applied to certain joints. Finally, the system iteratively adjusts the original motion to fit these constraints in the spacetime optimization process.
Instead of embedding fully physics-based model into our optimization framework, a simplified force response constraint is proposed, where we assume that the original and adjusted joint figures are both in a static equilibrium situation. We assemble springs at joints to exert muscle forces which counteract external forces. Without consuming time in extracting lots of physics parameters, we only have to project all muscle forces to the line of external force vector. By keeping each frame in static equilibrium, we can
simu
1.3
The
Kin Con
ulate the b
Flowch
y this exter
ollows:
rnal force
owchart
5
1. Divide the skeleton into different segments according to the loading BVH file.
2. Perform PCA on each segment respectively to transform the original moiton to a low dimensional space.
3. Embed the physical model in this low dimensional space.
4. Specify kinematic constraints and force constraints.
5. Adjust the original motion to fit constraints by spacetime optimization.
6
2 Related Works
It has been widely known that making character animation is a labor-intensive work. Instead of the traditional key-framing method which creates animation from sketch, Andrew Witkin and Michael Kass [9]
proposed spacetime constraints formulation that allows animators to control animation from a more high-level perspective. Recent years with the availability of realistic character animations by motion capture systems, there has been many researches regarding the reuse of captured data to lessen the cost of animation production. By combing the advantage of spacetime optimization, Zoran Popovic et al [5] took the original animation sequence as the underlying input, and then transformed it to a wide range of realistic character animations controlled by editing intuitive, high-level parameters.
Spacetime constraints approach provides a framework for creating character animation. The user first specifies what the character has to do by a set of kinematics constraints including pose constraints and mechanical constraints. Then the user specifies an objective function that defines how the motion should be performed such as minimizing energy consumption. In order to make the motion visually vivid, the physical structure of the character and the physics law form the dynamics constraints. Finally, an optimization algorithm solves the objective function to find out the motion trajectories satisfying kinematics and dynamics constraints. Thus this method has an intuitive control of the resulting motion.
7
However, the complexity of human musculoskeletal structure often leads the optimization problem to convergence difficulties. Zoran Popovic et al addressed this problem by mapping the original character structure to a simplified model with drastically reduced degrees of freedom (DOFs). Their entire algorithm breaks down to four stages.
Figure 2: Algorithm outline of motion transformation
First, character simplification creates an abstract character model containing the minimal number of DOFs necessary to capture the essence of the input motion. Second, find the solution of spacetime optimization that matches the motion of simplified character model at fitting stage. Third, adjust spacetime motion parameters, kinematics and dynamics constraints, or even objective function to edit the resulting motion. Finally, remap the editing change in motion onto the original motion to produce the final animation.
8
Fused body parts have redundant DOFs and the subtree of character hierarchy can be replaced with a single node in some cases of high-energy motion. By manually removing them, the simplified model not only captures the essence of the input motion without losing fundamental dynamics properties, but improves performance and facilitates convergence of the spacetime optimization.
Figure 3: Kinematic character simplification: (a) elbows and spine are abstracted away, (b) upper body reduced to the center of mass, (c) symmetric movement abstraction
Instead of simplifying character model directly, Safonova et al [7]
performed PCA to reduce the dimension of motion and solved spacetime optimization in a low-dimensional space. They exploited the observation of dynamic human motion having high degree of coordination. There is spatial coherence between body parts so that reduction of dimensionality is possible.
For the common human behaviors, they found that five to ten dimensions are sufficient.
9
Figure 4: Error between a full-dimensional motion and the corresponding k-dimensional representation
In their implementation, PCA were used to find basis vectors from a set of example motions. They solved the optimization problem to obtain the coefficients of linear combination of basis vectors that form the desired motion. Constraints were specified in the full-dimensional space and then projected onto the low-dimensional space. As long as the choice of example motion capture clips is similar behaviors to the desired motion, spacetime optimization in this approach can work well and effectively to generate natural-looking character animation.
Liu et al [3] proposed another approach that does not reduce the number of DOFs directly to improve spacetime optimization, but enforce linear and angular momentum of the motion to avoid heavy computation of complex dynamical model. They kept the pattern of linear and angular momentum as dynamics constraints by invariants and splines respectively.
10
Figure 5: The general angular and linear momentum pattern of a jumping motion
Furthermore, Liu et al [4] introduced more sophisticated dynamical model that incorporates several factors of locomotion derived from the biomechanical literature. This model accounts for passive joint forces due to muscles, tendons and ligaments, and simulated by springs and dampers. In order to overcome the difficulties of fine tuning of these physical parameters, they proposed a new algorithm, Nonlinear Inverse Optimization (NIO) to estimate the values from motion capture data. The underlying paradigm of NIO is a spacetime optimization problem that assumes the captured motion is optimal and then solves unknown parameters inversely. After that the user can adjust these parameters to generate new animations.
In contrast to long-horizon optimal plans, da Silva et al [6] proposed a short-term approach. They presented a controller, McSim, composed of a predictive component and a low gain proportional-derivative (PD) component.
The predictive component is a quadratic program (QP) with the linear dynamics model as constraints. It solves for the joint and external forces
11
which track the input motion for a short window of time into the future.
Then the PD component compensates for the errors generated by the predictive component due to high latency and modeling assumptions. McSim guides the simulated character dynamics toward input motion data at interactive rates while sacrifices optimality for computational performance.
Figure 6: An overview of McSim’s design
12
3 Methods
Our system first reads the human motion from a BVH file. BVH is a common file format used to store human motion. It arranges the articulation of human body in a hierarchy structure and specifies the motion data of each joint by three consecutive Euler angles, that is, rotation angle about Z axis, rotation angle about X axis and rotation angle about Y axis. Thus, there are 3 DOFs to describe a joint every frame.
Figure 7: Skeleton of human body
The above structure of human body has 18 joints. For a 5-second motion clip containing 165 frames, there are totally 8910 DOFs in this clip. If we treat each DOF as a variable in the optimization phase, such huge number of variables would result in a considerable computation time (even though this solution may not be an “optimial” solution because of local minimums) or a divergence situation. However, by utilizing spatial coherence and temporal coherence of human motion, we do not need to take all DOFs as variables in the optimization phase. In the following sections, details about how we reduce the number of variables would be introduced.
13
3.1 Segmentation
A human body has four limbs and a torso. Joints in the same part are more related than those in different parts because of the hierarchy structure.
For example, the movement of right shoulder would affect the moving of right wrist, whereas there is no direct relationship between right shoulder and left wrist. Thus we divide limbs and the torso into different segments in order to maximize this kind of local spatial coherence. Moreover, keeping local spatial coherence also provides more flexible controbility.
Figure 8: Each color for a segment
There are 5 segments. Each segment is divided empirically according to the physical structure of human body. They are the Torso, RightArm, LeftArm, RightLeg and LeftLeg respectively.
z Torso: there are 4 joints, including Hips, Chest, Neck and Head.
z RightArm: there are 4 joints, including RightCollar, RightShoulder, RightElbow and RightWrist.
z LeftArm: there are 4 joints, including LeftCollar, LeftShoulder, LeftElbow and LeftWrist.
z RightLeg: there are 3 jonts, including RightHip, RightKnee and RightAnkle.
z
there are 3 ntation, w g this kind
eft) Origin
3 joints, in e perform d of constr
nal motio efects by t kinemati
Effectors sitions of e raints.
on, (right)
LeftHip, Le n each seg n section 3.
matics c
ors at cert
d motion
z original m center of n is modifie
inal motio ed to react
on, (right
ints denisty of nal to its le ent size of traints. Li man body
cles in the t the condi
t) adjuste the user ca by combin
Figu cons
ure 11: (l straint
left) Origiinal motio
16
on, (rightt) adjusted motion satisfiedd the
17
3.3 Equilibrium-based Force Response
Instead of embedding complex dynamics to simulate human’s musculoskeletal model, we propose using the assumption of static equilibrium at each frame. This assumption take advantage of spacetime optimization such that we can only focus on the statics of each frame independently and let spacetime optimization keep the temporal connectivity between frames for us. In addition, by assembling 3 springs at X-axis, Y-axis and Z-axis respectively in the local coordinates for each joint, we do not need to induct novel variables as extra physics parameters in the optimization phase. To simulate the muscle force of some joint, we simply apply Hooke’s law for each spring of the joint.
Figure 12: A spring and a damper assembled at the joint
For some frame, the muscle force exterted by the joint i is made up of 3 component forces applied on local axes of the joint.
⎥ ⎥
18
By assuming that the original posture of human body at each frame is in static equilibrium, we do not need to calculate muslce forces when there is no external force; otherwise, the net force of total muscle forces and the external force must equal to zero. An external force is an additional force specified by the user to apply on some joint. It is a vector including magnitude and direction of the external force. Once the external force being added into the system, the whole human body is no longer in the state of static equilibrium.
As a result, angles of joints must be changed to exert muscle forces to counteract ths external force.
Every joint in the hierarchy would contribute 3 forces, each lying on the direction of its local axis. Then we transform these forces from their local coordinates to the world coordinates. Finally, all forces are projected onto the vector of the external force. Because the external force is set as a constraint, optimization process will drive the net force to zero.
G G External Force
Figure 13: Equilibrium θi
Joint i Fi = kiθi
θj
Fj = kjθj
External Force Joint j Gj
Gi
T external f constraint kind of con
n in the w d its neigh
ft) Origina plied to th
of force co
Constraint nstraint is whole anim
we can inc ch points t
traint onstraint i
e treat eac n is used t
tbor frame
al motion, he left limb
19
onstraints
s applied t mation wi crease the to negativ
is only ap ch frame to hold the
es.
, (right) ad b of body
that the
to the ent ill be influ e weight of
ve Y-axis f right arm
as the pe y this exte m by addin rmanent
frame in static sys een the ta
isfied the n our
20
3.4 Spacetime Optimization
The kernel of our system is a sequential quadratic programming (SQP) problem. We define the objective function companied with kinematics constraints and force constraints, and then solve our problem by the SQP solver SNOPT.
Such a motion optimization process is based on the original motion to ensure reality and also try to fit all user-specified constraints for motion editing. Thus the objective function is designed to preserve both smooth of the motion and similarity between the original motion and the result motion.
( ) ( )
The superscript j indicates the index of joint, and the subscript i indecates the index of frame. As a result, the first term in the above equation means that DOFs of the same joint between 2 consecutive frames should be close, while the second term means that the DOFs of joints in the target and original motion should be close as well.
21
In order to maximize accuracy and efficiency of the SQP solver, we provide the gradient of each DOF as follows,
( ) ( ) ( ) ( )
With sufficient information of gradients, computation time the SQP solver spending on finding out solutions can be reduced about 100 times. The number of iterations can also be decresed greatly to reach acceptable accurate results.
For all constraints, we append them to the objective function as soft constraints rather than hard constraints. The objective function becomes
( ) ( )
, where wc is the weight of constraint. It is a trade-off between accuracy and efficiency. We observed that little loss of accuracy can gain great time spent on trivial computation. At last, we refine the inaccuracy with Inverse Kinematics (IK).
22
3.5 Principle Components Analysis
To further improve the efficiency of spacetime optimization, we propose to use PCA to reduce the number of DOFs. Because PCA can preserve the most significant features or styles of data sets during transformation, in the situation of motion data it captures the major movement of body. When considering the hierarchy of human structure, the movings of joints which belong to the same body part are most correlated. Consequently, we perform PCA on each body segment independently such that less number of principle components can hold more accordant information without the bothering of unrelated data.
We arrange frame data as a n m matrix, where n is the number of original DOFs and m is the number of frames. After applying PCA, we can get the matrix of eigenvectors stored in column-major and the matrix of deviations as follows:
X VB Z
X: original data matrix, n by m V: matrix of eigenvectors, n by n B: matrix of deviations, n by m Z: matrix of z-scores, n by m
Eigenvectors in V are sorted according to their cumulative energy such that the eigenvector with the largest-magnitude eigenvalue is at the left.
We reduce the number of DOFs by keeping the eigenvectors of 98%
23
energy. It results in a submatrix W of V and a submatrix Y of B. The dimension of W is n n, where n is less than n, and the dimension of Y is n m. Finally, we use the elements of Y as the variables in optimization process instead of the original DOFs. In practice, n is usually half of n by preserving 98% energy because there is high coherence among joints in the same body segement.
At optimization, the object function is bascally uncanged except that variables are not original DOFs any more, the elements of sub deviations matrix Y instead. Then the gradient of each variable is slightly modified as follows:
, where summation means that all joints of the hierarchy described by the eigenvector related to x.
3.6 Spline Interpolation
To exploit temporal coherence between consecutive frames and further improve the quality of synthesis results, we use optimized key frames as control points of splines. Key frames are selected at regular intervals from the original motion. After applying PCA to their DOFs, the elements left in the matrix of deviations are set as variables in optimization process. To reconstruct the whole result motion, we simply interpolate other frames from these key frames.
24
4 Experiments and Results
There are 3 kinds of parameters that need to adjust manually. They are weight of continuity, weights of joints and force constants of joints, respectively.
4.1 Weight of Continuity
Weight of continuity influences the continuity between frames. The more weight of continuity, the more smooth of the motion. However, the constraints may not be satisfied accurately when keeping smooth. It results in a trade-off between smooth and accuracy. In practice, weight of continuity can be set heigher for environment obstacle constraints than for constraints of end effectors. Because the end effector does not need to hit a target precisely in the case of environment obstacle constraints, high weight of continuity can help the synthesis result look like more gracefully.
Figure 15: Relationshiop between accuracy and computation time
In Figure 15, it shows more computation time needed for higher
0 10 20 30 40 50 60 70 80 90 100
100 99 98 97 96 95 94 93 92 91 90 80 70 60 50
min (%) err (%)
accu
utes for 90 onds to c e is reduce frame clip hts of conti
ge of minu ed greatly t
takes abo
26
4.2 Weights of Joints
Joint Weight Joint Weight
Hips 4000 RightKnee 100
LeftHip 1000 RightAnkle 10
LeftKnee 100 Chest 2000
LeftAnkle 10 LeftCollar 1000
RightHip 1000 LeftShoulder 100
LeftElbow 10 RightElbow 10
LeftWrist 1 RightWrist 1
RightCollar 1000 Neck 10
RightShoulder 100 Head 1
Table 2
4.3 Force Constants of Joints
Joint Value of k Chest 4 Collar 8 Shoulder 8
Elbow 4 Wrist 2
Table 3
4.4 Results
z Original Motion
27
28
z Environment Obstacle Constraint
29
z Force Constraint
The external force is applied to the left limb of body.
30
31
5 Conclusions and Future Works
This thesis proposes a spacetime optimization based motion editing system which solves the configuration of joints satisfying user-specified kinematics and forces constraints. We exploit spatial and temporal coherence of human motion to save computation time while providing various kinds of constraints. With appropreiate set of kinematic constraints, QP solver can find acceptable solutions without tracking accurate muslce forces by a complex musculoskeletal model. To further expand the limits of kinematic
This thesis proposes a spacetime optimization based motion editing system which solves the configuration of joints satisfying user-specified kinematics and forces constraints. We exploit spatial and temporal coherence of human motion to save computation time while providing various kinds of constraints. With appropreiate set of kinematic constraints, QP solver can find acceptable solutions without tracking accurate muslce forces by a complex musculoskeletal model. To further expand the limits of kinematic