c) 
The following text and images have been copied from our paper "A Comprehensive Method for Geometrically Correct 3-D Reconstruction of Coronary Arteries by Fusion of Intravascular Ultrasound and Biplane Angiography" in Doi et al.: First International Workshop on Computer-Aided Diagnosis, September 20-23, 1998, Chicago IL © Elsevier Science B.V.
In the rapidly evolving field of intravascular ultrasound (IVUS), assessment of spatial structures still lacks a geometrically correct 3-D reconstruction. The IVUS frames are usually stacked up to form a straight vessel, neglecting curvature and the axial twisting of the catheter during the pullback. Quantification of this simplified data inevitably results in significantly distorted values. In this paper, we present a comprehensive method that combines the knowledge about vessel cross-sections obtained from IVUS with the knowledge about the vessel geometry derived from biplane angiography. The approach has been extensively validated in computer simulations as well as in phantoms and cadaveric pig hearts. An in-vivo application is under way. The algorithms showed good accuracy in determining the location and relative twist even in complex catheter paths and under noisy conditions.
The fusion of the two commonly used modalities for the assessment of coronary artery disease, angiography and intravascular ultrasound (IVUS), facilitates the combination of their advantages, and allows to overcome their well-known problems. Biplane angiography delivers accurate information about the vessel topology and shape, but the cross-sectional estimates are restricted to simple shapes like ellipses, and no information at all is provided concerning plaque or wall thickness. On the other hand, IVUS provides a high resolution assessment of plaque and vessel walls, but delivers no information about the location and especially the orientation of a specific IVUS frame in 3-D space.
c)
Figure 1: Acquired data from a right coronary artery of a cadaveric pig heart in a water bath; a) frontal and b) lateral angiographic projections with arrows indicating the tip of the inserted catheter, c) single IVUS image with segmented contours of the vessel wall.
Our method combines the information about vessel cross-sections obtained from IVUS with the information about the vessel geometry derived from biplane angiography. First, the catheter path is reconstructed from its biplane projections, resulting in a spatial model that already contains sufficient information to determine the location of the IVUS frames as well as their orientation relative to each other. Locations are derived from the time function, which implies the use of an automatic pullback device. Catheter twisting is calculated using a discrete version of the Frenet-Serret rules known from differential geometry. Our method uses the bending behavior of the catheter as reference for the absolute frame orientation along the pullback path. IVUS segmentation is performed with our previously developed algorithm.
Basis for the reconstruction of the catheter path are biplane angiograms as shown in Figure 1a, b. The catheter in its distal position (i.e. before pullback start) is extracted in both angiograms and reconstructed into 3-D afterwards. For the detection of the catheter, we use a dynamic programming approach, optimized for finding the local peak within the vessel profile that indicates the catheter location. The reconstruction process requires the knowledge about the imaging geometry; Figure 2a shows the common degrees of freedom provided by an angiographic device. Using the known 3-D locations of the X-ray sources and the angiograms, the reconstruction of a single point can be performed by calculating the corresponding projection rays, which should intersect in the original 3-D location of that point. Due to slight mismatches during the ray determination, they usually miss slightly, thus a generalized intersection point is defined for approximation (Fig. 2b). Systematics in those mismatches are analyzed at some reference points, and the imaging geometry, which is initially derived from the parameters as read from the angiographic device only, is refined to minimize the remaining error. Afterwards, the points along the catheter are matched between the projections and the 3-D path is reconstructed. The algorithm is described in detail in [1].
a) 
b) 
Figure 2: Geometric reconstruction from biplane angiograms; a) parameters of the imaging geometry, b) reconstruction of a single point P by approximation of the two points Q1 and Q2 marking the nearest distance of the projection rays.
For each IVUS image, a frame is generated from the 3-D trajectory. From the known pullback speed, which is assumed to be constant, the locations of the frames are assigned from distal to proximal along the reconstructed path. The location of the catheter in the IVUS images is known and usually in the center of the image. However, while the frame is always perpendicular to the catheter path, its axial orientation remains ambiguous. This problem is split up into two items: First, the relative orientation changes between adjacent frames are calculated, and in a separate step (described in section 2.4) the absolute orientation is determined.
The calculation of the relative twist has been described in detail in [2]. Using the three points of the catheter path next to the two adjacent frames, a circumscribing circle is calculated. The sector between the two frames is considered a part of the circle. Thus, the new frame is calculated by rotating the previous frame around the normal ni of the circle by the enclosed angle alphai (Fig. 3).
Figure 3: Sequential triangulation method for analytical determination of the relative orientation changes between adjacent IVUS frames.
Our method as earlier described in [3] automatically identifies the plaque/media interface (internal lamina, internal wall border), the media/adventitia interface (external lamina, external wall border), and the plaque/lumen interface (plaque border). Due to the use of a-priori information about 2-D and 3-D anatomy of coronary vessels and ultrasound imaging physics, the method can automatically determine vessel wall morphology and plaque volumes. In particular, to identify the position of the internal and external wall borders, the method searches for edge triplets representing the leading and trailing edges of the laminae echoes. A region of interest (ROI) has to be defined interactively in the first image of the sequence. Contextual information is used for the estimation of the ROI size and position changes in the successive frames. Anatomic knowledge about the coronary wall thickness is also used to constrain the search for the external and internal wall borders.
After the determination of the relative twist, and the mapping of the IVUS data in the frames using an arbitrary initial axial orientation, the absolute orientation has to be determined. This is done by calculating a correction angle, which is to be applied to all frames to minimize the reconstruction error. However, for this purpose a reliable reference is needed. As can be seen from Fig. 1, the catheter usually does not follow the center line of the vessel. Thus, the out-of-center location of the catheter is used to determine the axial angular errors.
From the angiograms, elliptical cross-section are reconstructed in each frame location. The out-of-center vector starts from the center of this ellipse and ends at the corresponding catheter point in 3-D. In the IVUS images, the vessel center is calculated as the centroid of the lumen contour, and is mapped into the respective 3-D frame. Within each frame, two values can be calculated: The difference angle between the angiographic and the IVUS out-of-center locations, and the length of the respective vector indicating the strength of the effect. The overall correction angle is derived as the weighted mean of the difference angles. Since slight reconstruction errors may occur, the local tolerances are calculated as the standard deviation of the difference angles within a moving fixed-size environment, and used as negative weight [4].
The fusion system has been validated in several computer simulations, phantom studies, and data from cadaveric pig hearts. Figure 4 shows an example from the set of computer simulations; a very tortuous 3-D path was generated from two sine waves of different frequency, where the first one controls the axis of rotation (i.e. bending direction), and the second wave controls the amount of rotation (i.e. bending strength). At ten locations along the entire path of 100 mm in length, 2-D coordinate systems were calculated that indicate the orientations of the respective IVUS frames. To simulate the angiographic resolution, the 3-D path was discretized to straight segments of 500 µm in length. The algorithms for determination of the relative and absolute orientations were applied to this path and showed a root-mean-square (RMS) error of 1.05° with a maximum of 2.52° at the ten reference frames.
a,b) 
c) 
Figure 4: Computer simulation of an extremely tortuous path derived from two sine waves with different frequencies; a) detail and b) full path, c) calculated axial twist after discretization of the path to angiographic resolution.
Figure 5: Visualization of the data shown in Fig. 1 as an automatically generated VRML-2.0 model; the angiographic setup is shown as well as selected IVUS images and the triangulated results of the segmented contours.
In the cadaveric pig hearts, eight paper clips were implanted in each of two right coronary arteries (RCA), so that they were visible in both angiograms and IVUS images. These artificial landmarks were used to determine the effective pullback speed as well as the correct orientation of the IVUS image. Three manual pullbacks were performed in each of the arteries over a length of 110-130 mm with a desired pullback speed of 1.0 mm/s. However, the effective pullback speed was 1.14±0.34 mm/s over all six pullbacks, and the mean axial reconstruction error was 21.96±4.87°.
From another RCA of a cadaveric pig heart as shown in Fig. 1, the error in the absolute axial orientation was determined without artificial landmarks. The algorithm reliably found the required correction angle and distributed the weighting factors in a proper way. However, areas with a high out-of-center location of the catheter but with errors in the longitudinal location represented a conflict. These errors probably resulted from inhomogeneities in the manual pullback.
Our system provides high accuracy in 3-D reconstruction of the vessel topology, and a spatially correct assignment of plaque and wall data as delivered by the IVUS segmentation. Thus, highly accurate volumetric quantifications are feasible, as well as geometrically accurate and realistic visualizations of the vessel using virtual reality techniques (Fig. 5). However, the errors introduced by the manual pullback and especially from the phase shifts in the mechanically driven catheter system could only partially be adjusted. While the localization error can be minimized by using automated pullback devices, further effort has to be devoted to eliminiate the axial errors in the catheter systems and/or to detect and to correct them.
This work has been supported in part by grants Pr 507/1-2 and Wa 1280/1-1 of the Deutsche Forschungsgemeinschaft, Germany, and by grants IA-94-GS-65 and IA-96-GS-42 of the American Heart Association, Iowa Affiliate.