In this paper, we seek to fit a model, specified in terms of connected ellipses, to an image silhouette. Some algorithms that have attempted this problem are sensitive to initial guesses and also may converge to a wrong solution when they attempt to minimize the objective function for the entire ellipse structure in one step. We present an algorithm that overcomes these issues. Our first step is to temporarily ignore the connections, and refine the initial guess using unconstrained Expectation-Maximization (EM) for mixture Gaussian densities. Then the ellipses are reconnected linearly. Lastly, we apply the Levenberg-Marquardt algorithm to fine-tune the ellipse shapes to best align with the contour. The fitting is achieved in a hierarchical manner based upon the joints of the model. Experiments show that our algorithm can robustly fit a complex ellipse structure to a corresponding shape for several applications.