Journal of King Saud University - Computer and Information Sciences (2015) xxx, xxx-xxx

King Saud University

Journal of King Saud University -Computer and Information Sciences

www.ksu.edu.sa www.sciencedirect.com

Journal of

King Saud University -

Computer and

Information Sciences

NURBS-based geometric inverse reconstruction of free-form shapes

Deepika Saini *, Sanjeev Kumar, Tilak Raj Gulati

Department of Mathematics, Indian Institute of Technology Roorkee, Roorkee 247667 India Received 17 June 2014; revised 18 November 2014; accepted 14 December 2014

KEYWORDS

3D reconstruction; Free-form shapes; Non-Uniform Rational B-Spline (NURBS); Perspective projection; Quadratic programing problem (QPP)

Abstract In this study, a geometric inverse algorithm is proposed for reconstructing free-form shapes (curves and surfaces) in space from their arbitrary perspective images using a NonUniform Rational B-Spline (NURBS) model. In particular, NURBS model is used to recover information about the required shape in space. An optimization problem is formulated to fit the NURBS to the digitized data in the images. Control points and weights are treated as the decision variables in the optimization process. The 3D shape reconstruction problem is reduced to a problem that comprises stereo reconstruction of control points and computation of the corresponding weights. The correspondence between the control points in the two images is obtained using a third image. The performance of the proposed algorithm was validated by taking several examples based on the synthetic and real images. Comparisons were made with a point-based method in terms of various types of errors.

© 2015 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction

The space curves play a crucial role in the construction and designing of all types of objects in the real world. The basic curves (lines and conics) are easy to interpret in terms of algebraic equations. However, such an interpretation is not easy in the case of the free-form shapes (having a helical shape, the

* Corresponding author. Tel.: +91 01332 285824. E-mail addresses: sainddma@iitr.ac.in (D. Saini), malikfma@iitr.ac. in (S. Kumar), trgorfma@iitr.ac.in (T.R. Gulati). Peer review under responsibility of King Saud University.

form of some biological objects, edges of some handcrafts, turbine and aircraft blades etc.). Nowadays, all disciplines of engineering design and graphics community are widely used in free-form shaping. In such systems, the Non-Uniform Rational B-Spline (NURBS) (Piegel and Tiller, 1995) is used to represent a free-form shape. Some of the reasons behind this widespread acceptance of the NURBS are as follows:

• NURBS represents a mathematical form, that addresses the problem of analytical shapes, e.g. lines, conics etc., as well as the free-form shapes. Therefore, basic and the freeform shapes, both are represented precisely by the NURBS.

• NURBS is shape invariant under affine and perspective projections.

• NURBS is the generalized form of B-spline and Bezier

http://dx.doi.org/10.1016/j.jksuci.2014.12.010

1319-1578 © 2015 The Authors. Production and hosting by Elsevier B.V. on behalf of King Saud University. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

ARTICLE IN PRESS

2 D. Saini et al.

Figure 1 Example of NURBS curve: a 2D curve with five control points and a complex helix drawn with only thirteen control points.

• NURBS provides the flexibility to design the large variety of shapes. It may be done just by adjusting the control points and their corresponding weights.

One of the challenging problems in computer vision is to recover the 3D object using its one or more perspective views due to the following reasons:

• Free-form shapes are not always planar.

• These shapes are not invariant under perspective projection.

• Their shapes are more complex than planar ones.

• Multiple correspondences exist in their different perspective views.

The main contribution of the proposed study is toward a compressive reconstruction of the free-form shapes from their arbitrary perspective projections. In particular, the proposed algorithm consists of the following main steps:

(i) A two-step linear process for fitting the NURBS from the digitized data in perspective views.

(ii) Establishing a correspondence between the control points of NURBS using a third view.

(iii) Reconstruction of the required free-form NURBS shape in the 3D space by estimating control points and their corresponding weights from the fitted 2D NURBS shapes.

The presented study is the extension of our previous work (Saini and Kumar, 2014). We have extended our ideas for the free-form curves as well as for the surfaces.

The proposed approach has many advantages over the standard triangulation-based techniques (Faugeras, 1993).

• Here, the reconstruction may be done in a general stereo framework instead of a particular model (rectified stereo setup).

• Computations of the control points and their corresponding weights give a better approximation.

• Moreover, the pinhole camera models are considered using the invariant property of NURBS under perspective projection.

• Reconstruction of the control points reduces the complexity of the proposed algorithm tremendously when compared to standard point-based triangulation approaches (Faugeras, 1993; Xiao and Li, 2005).

Finally, several experimental results are given to validate the proposed approach.

2. Prior work

In this section, we summarized the existing work in literature related to the reconstruction of free-form shapes in space. The area of reconstruction of the 3D free-form shapes is continuously progressing as an application of projective geometry in computer vision and graphics. However, most of the work in the existing literature was focused on the reconstruction of lines and conics (Balasubramanian et al., 2002, 2003; Kanatani and Liu, 1993; Kumar et al., 2006; Quan, 1996; Sukavanam et al., 2007). The existing work in this direction was grounded on the basis of point correspondences or analytical formulation. Xie (1994) and Xie and Thonnat (1996) proposed an analytical formulation-based approach for reconstructing quadratic curves. However, the problem of unique reconstruction was not solved by their method. Balasubramanian et al. (2002, 2003) proposed an approach for the unique reconstruction of quadratic curves from three perspective views. First, they found a solution for the required curve from the intersection of lines and surface (cone), and then used a third view for determining the unique reconstruction. Kanatani and Liu (1993) developed a computational procedure for inferring the geometry of conics in space using the orthogonal projections. Kumar et al. (2006) and Sukavanam et al. (2007) proposed an approach for the unique reconstruction of algebraic curves by using the intersection of geometric entities. The beauty of their presented method is the applicability in the reconstruction of planar as well as non-planar curves, while Balasubramanian et al. (2002) and Balasubramanian et al. (2003) were used only for the planar curves. Quan (1996) resolved the matching problem for conics using two perspective views. In his approach, a unique solution was also discussed by using the non-transparency constraint related to the

conics. Kahl and Heyden (1998) shown that the epipolar geometry can be interpreted in terms of the fundamental or essential matrix for the corresponding conics in two images. Wong and Chung (2003) discovered a way of matching the objects, and reconstructing the 3D structure using its two views. In their method, some topological relationships between the object points and the planarity constraint of object surface were used.

A closed form solution for reconstructing the 3D algebraic curves was proposed by Ma (1993). However, this approach was applicable only for the planar curves. In case of non-planar curves, this approach was switched to shape-from-motion instead of stereo. In An and Stereo (1996), an efficient reconstruction was proposed for the planar curves. The algorithm presented by Peng et al. (2012) used a sequence of the input images for the reconstruction process. The presented algorithm was actually based on a corollary related to the projective depth-map, which states that for any pair of the 3D points, depth can be apprehended as a linear factor. Yang et al. (2005) proposed a new formulation to get the 2D

symmetric patterns in the 3D space from its calibrated perspective images. Kahl and August (2003) proposed a method for finding the correspondence and recovering the 3D curves using multiple images.

Many applications require surface reconstructions that are applicable in various fields e.g., reverse engineering techniques of CAD models, computer animation etc. (Koch et al., 2000; Salman and Mansor, 2013). Salman and Mansor (2013) proposed an investigation of reverse engineering technique to generate the free-form surface models in CAD systems from physical models. Weng et al. (1988) developed a closed form solution for the planar reconstruction from two views. A stereoscopic scene analysis system for modeling the 3D objects from its sequence of stereo images was presented in Koch (1995). An automatic reconstruction scheme was presented in Koch et al. (2000) for the modeling of 3D objects from a set of the input images. This automatic scheme did not use any prior information of the required object except its rigidity. Shu and Roth (2003) presented a method to recover the 3D location of a texture-less surface

Table 1 Equations and various parameters used in the generation of synthetic curves and surfaces for experimental study.

Experiment No. Generating equations Range Points used

1 (Fig. 4) ' X = 2 cos t t 2 [0,if] 31

Y = 2 sin t

s Z = 2(t + 1)

2 (Fig. 6) ' X = sin t t 2 [0, 3p] 61

Y = cos t

Z = 1 + t

3 (Fig. 7) ' X = linspace(—2,2,11) 121

Y = linspace(—2,2,11)

(XY) = meshgrid(X, Y)

Z _ 1 Z _ (1+X2+Y2)

>surf(X, Y, Z)

4 (Fig. 8) ' X = linspace(—2,2,20) 400

Y = linspace(—2,2,20)

(XY) = meshgrid(X, Y)

Z = sin( V X2 + Y2)

>surf(X, Y, Z)

using less number of images. After acquiring the 3D points, B-spline surface fitting was used to get the smooth surface. In Zeng et al. (2004), an alternative paradigm for the reconstruction was proposed, where the sparse 3D data and the 2D calibrated views were used in the reconstruction of dense surface. To perform this procedure, small surface patches were constructed from the input images, and then these patches were propagated in their neighborhoods to get the dense surface. Another approach for shape reconstruction was

presented in Kumar and Kumar (2012) using the hybridization of shape from shading and stereo.

In the reconstruction of free-form shapes in the 3D space, the above mentioned methods have not always worked. NURBS (Piegel and Tiller, 1995) has several properties that make it the most suitable candidate for representing any shape or structure in the 2D and 3D geometry. Moreover, it can be determined completely by its control points in 2D and 3D cases. It implies that the problem of curve or surface

Table 2 Projection matrices used to project synthetic data in two image planes in various experiments. Experiment No. TL

11 0 0 -20

0 1 0 0

0 0 1 0

( 1 0 0 -10

0 1 0 0

0 0 1 0

(1 0 0 -30

0 1 0 0

0 0 1 0

(1 0 0 -30

0 1 0 0

0 0 1 1

10 0 0 0 10 0 0 0 11

10 0 0 0 10 0 0 0 10

10 0 0 0 10 0 0 0 11

10 0 0 0 10 0 0 0 11

4(b) and (c)

6(b) and (c)

7(b) and (c)

8(b) and (c)

Table 3 Knot sequence used in various experiments.

Experiment No. Knot sequence

1 u-knots = [0 0 0 if5 238 1 1 1]

2 u-knots = p^i.LJo^îôtôïôtô1111]

3 f u-knots =[0 0 0 1 3 1 1 1]

v-knots =[0 0 0 § § 1 1 1] u-knots =[0 0 0 1 i § 4 4 4 4 1 1 1] v-knots =[0 00t-3tffl11]

Wire u-lmote = [0 000ltîl1îl2bl30tol6btT>töt01111]

Vessel f u-knots =[0 0 0 0 112441111]

v-knots =[0 000 3322 1 1 1]

reconstruction in the 3D space may be treated as the reconstruction of the sparse set of control points of the NURBS representation of the corresponding curve and surface.

To utilize the advantages of the above mentioned properties of the NURBS, very few efforts were made in the direction of the 3D reconstruction of the free-form curves and surfaces. Ding et al. (2003) proposed a method for reconstructing free-form space curves from their stereo views under the affine

projection model. However, the obtained results demonstrate a possibility to use splines for the space curve reconstruction under the projective model. They used the standard least-square fitting (Rogers and Fog, 1989) to approximate the digitized data in the orthographic views to a B-spline model. As the problem of 3D reconstruction is a geometrically inverse problem and hence ill-posed, therefore a small error in approximation can cause a large deviation in the final reconstruction.

Figure 4 Reconstruction result of a set of 31 data points in experiment no. 1: (a) 3D initial curve; (b) 2D projection of initial curve onto the left image plane; (c) 2D projection of initial curve onto the right image plane; (d) 3D reconstructed curve; (e) projection of the original and reconstructed points in the third view.

Figure 5 Reconstruction result of a set of 31 data points in experiment no. 1: (a) point-based reconstruction approach; (b) proposed reconstruction approach.

Figure 6 Reconstruction result of a set of 61 data points in experiment no. 2: The equation of circular helix is considered to be xw = sin(t), yw = cos(t) and zw = 1 + t, where t = 0 : p/20 : 3p (a) 3D initial curve; (b) 2D projection of initial curve onto the left image plane; (c) 2D projection of initial curve onto the right image plane; (d) 3D reconstructed curve; (e) projection of the original and reconstructed points in the third view.

Later, Xiao and Li (2005) proposed an optimized framework for the fitting of NURBS curves in perspective views for a given digitized data set. They used the epipolar constraints (Faugeras, 1993) in stereovision framework, and for simplifica-

tion considered only a rectified stereo system. Siddiqui and Sclaroff (2001) proposed a method for reconstructing the 3D surface from multiple views using rational B-splines. They exploited the properties of the projective invariance of rational

Figure 7 Reconstruction result of a set of 121 data points in experiment no. 3: (a) 3D initial surface; (b) 2D projection of initial surface onto the left image plane; (c) 2D projection of initial surface onto the right image plane; (d) projection of the original and reconstructed points in the third view; (e) 3D reconstructed surface.

B-splines to gain a solution of the 3D surface reconstruction problem. A new way of inferring and reconstructing the 3D objects was presented by Lavoie et al. (2007). This reconstruction was based on finding the unique control points in the pro-jective images and evaluating the associated NURBS curve. A stereo vision method with the NURBS curve was also presented recently by Zhang et al. (2013) to achieve the 3D reconstruction of a bent tube. In their approach, the NURBS curve was chosen as the primitive of the reconstruction. For the curve reconstruction problem, Zhang (2013) presented a method in which the NURBS was employed for the curve representation in the 3D space. This method is applicable for the general curves as well as for the planar curves.

3. Preliminaries

In this section, some standard definitions and terminologies, that are used in the subsequent sections, are mentioned.

3.1. NURBS curve

A rational B-spline curve (Farin, 1992; Piegel, 1991), C(u) : R3 ! R is given as:

C(u) = P,R,.k(u),

where, P represents the ith control point of the space curve C(u). The rational B-spline basis function is given as:

Ri,k(u) =:

W,Bhk(u) E^WBk (u) '

i = 1, ...N + 1,

where, Wi is the weight that corresponds to the control point Pi and Bik(u) is the normalized B-spline basis function of degree k, which is defined recursively as follows:

1 if Xi 6 u < xi+1, 0 if otherwise.

Bi,k(u) = Bi,k_1 (u) + Xi+k+1 " Bi+1, k_1 (u) , (4)

xi+k xi

Xi+k+1 _ Xi+1

where, xt denotes the knot that forms a knot sequence n = {xi, x2 ,..., xN+k+i} with Xi 6 xi+1 and u denotes the independent variable of the basis functions. When the knot sequence is not restricted to be uniformly spaced, the rational B-spline curve is called NURBS. The degree k, number of knots m, and number of control points N + 1 are related by

ARTICLE IN PRESS

8 D. Saini et al.

projected control points

(d) (e)

Figure 8 Reconstruction result of a set of 400 data points in experiment no. 4: (a) 3D initial surface; (b) 2D projection of initial surface onto the left image plane; (c) 2D projection of initial surface onto the right image plane; (d) projection of the original and reconstructed points in the third view; (e) 3D reconstructed surface.

the formula m = N + k + 1. For NURBS, the knot vector takes the form n = {a, a,..., a, xk+u ..., xm_k_u b, •••, b, bg, where the end knots (a and b) are repeated with multiplicity k + 1. In most practical applications a = 0 and b = 1, as is assumed throughout this paper. Example of NURBS curves generated with five control points and a complex helix drawn with only thirteen control points are shown in Fig. 1.

3.2. NURBS Surface

A NURBS surface (Farin, 1992), defined over the parametric variables u and v is given as:

Q(u, v) = £ v),

j=1 j=1

where, Pi;j represents a 3D polygon net-points of the surface Q(u; v). The bivariate rational B-spline basis function Sij (u, v) is given by:

WjB^ujv)

Sij {u, v) n+1 v->M+1 „, D / \D /

E,=1 E;=1 W,JB„k(u)BJ,l(v)

i = 1,..., N + 1; j = 1,..., M + 1,

where, Bik(u) and B,-/(v) are the normalized B-spline basis functions of degrees k and l, respectively. The NURBS surface has O(NM) control points P^ with their corresponding weights Wij. The number of the control points M + 1 and N + 1 are defined by the surface order in each parametric direction u and v, respectively.

3.3. Point-based reconstruction method

Point-based correspondence approach (Faugeras, 1993) is a well studied problem with a long history in the field of traditional stereo reconstruction. In this method, from each pair of corresponding image points, captured with two or more different view points, the corresponding 3D point is reconstructed using triangulation (Xiao and Li, 2005). However, such type of correspondence search is itself a challenging problem. Main reason is lying in its large searching space which also get affected by various factors e.g., noise, focal length distortion, etc. Huge efforts were made to solve the point-based reconstruction problem over the past two or three decades, yet the problem is far from the exact solution due to its ill-posed nature (Weng et al., 1988). Moreover, point-based reconstruction approach does not use the structural information of object

surfaces which also increase the difficulties in reconstruction procedure.

To overcome the difficulties faced in point-based reconstruction approaches, different geometric shapes (1D and 2D) are used for the efficient 3D reconstruction. The 2D shapes are considered as surface patches (Zeng et al., 2004) whereas the 1D primitives are usually taken as lines and curves (Farin, 1992; Piegel, 1991). The 1D primitives or features are more beneficial than the 2D shapes because of providing the more information cues. In the case where only image points are taken as the primitives, the point-based correspondence search must be carried out. This search is a major error-prone problem that affects the quality of reconstructed images.

4. Proposed reconstruction method

The proposed reconstruction algorithm relies on the following three main steps: (1) NURBS fitting; (2) Stereo correspondence between the control points; (3) Weights computation. The following texts in this section describe each of these three in detail.

4.1. NURBS fitting

Fitting of curves and surfaces from the given data is a well known problem in various domains of Science and Engineering. In general, fitting is obtained by the minimization of an error function for the input data points. The type of error, i.e., norms used for the error minimization include l1 , 12, and fc. The choice of appropriate norm depends on the physical domain of the problem. From the mathematical point of view, such a problem leads us to an over-determined system of linear equations (Heidrich et al., 1996).

A number of methods were suggested for fitting of the NURBS curves and surfaces from the given data points. Most of them used the concept of least-square fitting (Ma and Kruth, 1995). The main steps employed in such techniques are the parameterization of measured points and knots, followed by the least-square minimization. Apart from this, various nonlinear optimization techniques were used from quadratic programing (Luenberger, 1973) to evolutionary computation approaches (Zhang et al., 2009). The nonlinear optimization approaches are good at fitting accuracy, but need more computational time. Some of the techniques are based on the interpolation of the given data (Farin, 1992; Piegel, 1991). However, in the case of NURBS, these approaches are not applicable to practical applications due to the requirement of weight vectors. A two-step linear approach was proposed for NURBS fitting by calculating the weights and control points in a separate process with a linear optimization process (Ma and Kruth, 1998). The method employed in the presented study is inspired by the two-step linear approach. Fitting of the NURBS curves or surfaces consists of the following steps:

4.1.1. Computation of weights

According to the definition of NURBS (Eq. (5)) and its geometric invariance property (Piegel, 1991), let q(u , v) be a projection map of NURBS surface Q(u, v) of degree (k,1) with O(NM) control points as:

q(u, v) =

EJ EjJ wlJp,Jsl,k(u)Bj,i(v)

Em1 wyBkMB,/(v) '

The evaluation of such NURBS surface at n1 parameter values {ui, u2 ,... ,un1} along the u-direction and n2 parametric values {vi, v2,... ,vn2} in v-direction, yields the following equations:

ES1 E^WjpyAfcK )Bj,i(vt2)

q'l, t2 N+1 ^M+1 D , in/ i '

Et i E,=i wijBi,k(uti)Bj,i(vt2)

ti = l,... ,Ui t2 = l,...

Let B be the matrix containing B-spline basis functions. In a compact form, Eq. (8) can be written as:

Dx ■ B ■ W — B ■ Px Dy ■ B ■ W — B ■ Py;

where,

px — [p^H . . . ,pN+1 ,M+ 1WN+1 ,M+ 1]T ; py — [py,1 w1,1, . . . ,pN+1 ,M+1 wN+1 ,M+ 1]T, W — [W1,1, ... ,Wn+ 1 ,M+ 1]T ,

Dx— diag{qX1;... ,qnX1 ,n2} ;

Dy— ^gKn... ;q„y1 ,n2}-

Eq. (9) can be rewritten in the following system of linear equations:

B 0 0 B

DxB DyB

After simplification, Eq. (12) can be written as:

BTB 0 0 BTB 0 0

BTDxB BTDyB

Px 0 ■

where, M = Mx + My is a symmetric and non-negative matrix with

Mx = BTD2xB - (BTDxB)(BTB) 1(BTDxB), My = BtD2B - (BTDyB)(BTB)-1 (BTDyB).

The third row of the system given in Eq. (13) is used to identify the weights, i.e.,

M W 0.

Use of the least-square norm for solving such a system was suggested in Ma and Kruth (1998). But this minimization process does not give any guarantee about positive weights. In the presented study, a quadratic programing (QP)-based algorithm is adopted to find the set of positive weights (Boot, 1964; Luenberger, 1973). The formulated minimization problem takes the following form:

Minimize WTMTMW subject to W > 0.

The optimal solution of the above QPP (Quadratic Programing Problem) gives a guarantee of positive weights, which is a necessary in of NURBS fitting.

4.1.2. Computation of control points

Once we computed the weights, the control points can be obtained from Eq. (9) by substituting the values of W. We used this method for fitting the NURBS curve and surface in both of the image planes which gives the set of control points and their corresponding weight vectors. Fitting of the NURBS curve is not substantially different from the fitting of a NURBS surface. The same method is applicable for the NURBS curve fitting of the digitized data.

To demonstrate this fitting strategy, a NURBS curve is fitted from 41 data points having order 4, and 13 control points. The equation of this space curve (ellipse) is considered as x = 5 + 3 cos(t) _ 2sin(t), y = 4 _ 2cos(t) + 3 sin(t) and z = 0.82 _ 0.02 cos(t) _ 0.02 sin(t) where t varies from 0 to 2p. This curve is projected onto the left image plane using a projection matrix given as:

10 0 0 0 10 0 0 0 11

sequence

[0 0 0 0 10 10 tl 10 150 160 180 19 190 1 1 1 1]. Then, B-spline basis

10 10 10 10 10 10 10 10 matrix is given by:

1 0 0 0.4219 0.4961 0.0794

0 0 0 ■■■ and the matrix M is given as:

0.0222 0.0601 0.0746

-0.0601 0.0746

0.0001

0.1736 0.2386

-0.2386 0.3784

0.0003 0.0007

0.0001 -0.0003 0.0007

0.0188

On solving the QPP (Eq. (16)), the following weight vector is obtained:

(i) Pixel locations in the left (first) image 11.

(ii) Pixel locations in the right (second) image 12.

(iii) Pixel locations in the third view plane (additional constraint for uniqueness) 13.

(iv) Parameters related to the imaging device and its perspective geometry: f u/2,xd,yd,zd and (a,-, b, c,-) for i = 1,2, 3.

Let P(xw, yw, zw) be a control point in the NURBS representation of the required shape in 3D space, and p1(X1, Y1) is its image in the first camera. From the perspective invariance property of NURBS, p1 must be a control point of the NURBS representation of the image of the required 3D curve or surface in the first camera. The equation of the line O1 p1 through the center of projection (COP) of the first camera O1(0,0,0) and the point p1 is given as:

xw _ yw _ zw _ k

X — Y — f — '.

Moreover, the collinearity equations (Balasubramanian et al., 2002; Balasubramanian et al., 2003) for these points are given

X1 = /1 xï, Y = /1 yw.

Similarly, take the pointp2(X2, Y2) as image of P(xw,yw,zw) in the second camera. The equation of the line O2p2 passing through the COP of the second camera O2 and the control point p2 can be easily written as:

X2 — /2

2 J 2 (x, —

(Xw-Xd) cos «1+(yw

_ / (Xw-Xd) cos «2 + (yw

2 (Xw-Xd) cos «3 + (yw

-yd) cos b1+(zw-zd) cos Ç1 -yd) cos b3+(zw-zd) cos y3 '

-yd) cos b2 + (zw zd) cos Ç2

-yd) cos b3 + (zw zd) cos Ï3 '

From these two 2D points p^Xi, Yi) and p2(X2 , Y2), the 3D point P(xw yw zw) can be reconstructed by the intersection of the line O1p1 and O2p2. In this manner, we obtain all the control points that are required for the NURBS representation of the 3D curve or surface. For the uniqueness of this reconstructed 3D NURBS curve or surface, a third view is used which is described in the following subsection.

W — [0.9984 1.0067 0.9994 0.9912 0.9965 1.0088 1.0148 1.0118 1.0013 0.9893 0.9868 1.0009 1.0113]1x13

and then, the following control points Pl are calculated using Eq. (9):

10.0000 9.5342 8.0096 4.9791 2.4999 3.2936 5.2701 8.0633

2.4464 1.3955 2.1413 4.3203 7.1358 9.6140 10.8467 10.4783 10.0000 9.5623 9.2250 7.2812 4.5138 1.9025 0.3547 0.4679 1.6908 2.5001

4.2. Detailed process of reconstruction of free-form shape in 3D space

The imaging setup for the reconstruction process is shown in Fig. 2. In order to reconstruct the free-form shape in 3D space from its arbitrary perspective views, following inputs are needed:

4.3. Correspondence between control points

For the unique reconstruction of the 3D curve, establishing the correspondence between the control points in two views is needed. For this, an additional constraint called third view is used. Let 13 be the third image that consists of the set of pixel coordinates (X3i , Y3i). Here, again we use a two-step linear

approach for NURBS fitting and obtain the control points in Using triangulation (Sections 4.2 and 4.3), following points

the third view. Suppose both the image planes I1 and 12 have which control the shape of the required curve (ellipse) in space Ni+1 and Nj+1 control points, respectively. Here, i = 1,..., n1 are obtained:

"8.0000 7.5861 6.3318 3.9565 1.9798 1.1515 1.7970 3.6676 6.0445 8.0220 8.8471 8.4225 8.0000" 2.0000 2.6229 4.1607 6.3887 7.7074 7.6127 6.1332 3.8403 1.6098 0.2926 0.3863 1.3672 2.0001 0.8000 0.7958 0.7901 0.7931 0.8063 0.8247 0.8414 0.8498 0.8469 0.8337 0.8153 0.8042 0.8000

and j — 1,... , n2 and n1, n2 belong to the set of natural number. Select two control points arbitrarily, one from each (first and second image planes), and find out the corresponding 3D point. Then, this reconstructed point in space is projected onto the third image plane 13. If the projected control point matches any of the control points in the third image plane with some threshold, this implies the correspondence of the two control points in the respective image planes, if not, take another control point and repeat the above procedure for all the control points. Once the correspondence is established between any pair of the control points, continue the search. This procedure is repeated for all the control points. For computational simulation, some threshold values (e) are considered.

The surface reconstruction algorithm is explained in the Algorithm 1.

Algorithm 1. : Surface reconstruction Algorithm

The same example of space curve (ellipse) is taken to demonstrate the procedure of 3D reconstruction that was used for NURBS curve fitting earlier. The parameters of the imaging setup are taken as: xd = 20 , yd = 0 , zd = 0. Following control points are obtained in the second (right) image plane:

The graphical illustration of the original and the reconstructed control points of the space curve (ellipse) is shown in the Fig. 3. For the quantitative analysis, mean, maximum and standard deviation (SD) values with the proposed algorithm are calculated, and also compared with the classical point-based approach (Xiao and Li, 2005). These values are found 0.0024 & 0.2185 ,0.0155 & 0.3700 and 0.0033 & 0.2090, respectively (values are given in the proposed & point-based approach manner). Thus the proposed algorithm reconstructs the shape quite accurately.

5. Results and discussions

5.1. Qualitative results

The proposed methodology was implemented to reconstruct various free-form shapes (curves and surfaces). For the experimental study, real as well as synthetic images were used as the inputs of the algorithm. The code for the reconstruction algorithm was developed in MATLAB and a computer with 2.53 GHz Intel(R) 13 processor with 3.0 GB RAM was used for computation. Details about generating the input data are given in Table 1, whereas the knot sequence for each experiment is given in Table 3. The original 3D plots of all these synthetic shapes are shown in Figs. 4-8(a). The whole process for carrying out the reconstruction results is mentioned as follows:

1. Generate the digitized data points in both of the views by projecting the synthetic 3D data points with the projection matrices (as given in Table 2).

2. Perform NURBS fitting to these projected data points in both of the views to obtain the control points. The fitted NURBS curves and surfaces are shown in Figs. 4-8(b) and (c).

3. Establish the correspondence between the control points in the two images using a third view (Figs. 4-6(e) and Figs. 7 and 8(d)).

4. Reconstruct the control points of the NURBS representation of the required shape (curves or surfaces) using triangulation. Moreover, assign the appropriate weights to each reconstructed 3D control point to generate the complete NURBS curves and surfaces in space. These are shown in Figs. 4, reffig5, 6(d) and Figs. 7 and 8(e).

Input: Data points Dx and Dy in both image planes Parameters: Knot vectors and degree of B-spline basis functions in both parametric directions

Compute normalized B-spline basis function B1 in one parametric direction and B2 in another direction for i = 1 to n1 do for j = 1 to n2 do

B = B(i j) = B1 x B2 end for end for

Compute M using Eq. (14)

Compute weights W from M ■ W = 0 using Quadratic

n ui i Minimize WTMTMW Programing Problem

subject to W > 0 Compute control points P in both image planes using Dx B W B Px

Establish correspondence using the third perspective view Reconstruct control points in 3D using triangulation and weight computation

Dy ■ B ■ W — B ■ Py

114.9999 -116.1383 -118.5541 -121.1030 -121.5710 -119.8463 -116.7098 -113.3530 ■■■ -115.0001" 2.5000 3.3061 5.2758 8.0585 9.5612 9.2258 7.2852 4.5187 2.5000

(d) (e)

Figure 9 Curve reconstruction result of real stereo images: (a and b) real stereo images of a wire; (c and d) results of NURBS fitting with optimized control points; (e) 3D reconstructed curve.

It can be observed from Figs. 4-8 that the proposed algorithm gives acceptable reconstruction shape in all cases. Not only for the curves, but the proposed algorithm is equally effective in the reconstruction of surfaces. Thus, all the experiments give a proper qualitative validation to the proposed reconstruction methodology. Using the standard point-based reconstruction approach (Faugeras, 1993), the reconstructed 3D curve is shown in Fig. 5(a). Fig. 5(a) shows a linear interpolation of these reconstructed points, which makes it easy to visualize the recovered shape of the curve. The recovered 3D curve is neither smooth nor accurate. In contrast, with the proposed algorithm, the resulting figure is shown in Fig. 5(b).

Figs. 9 and 10 show the reconstruction results based on the real data. In both of the stereo pairs, images were acquired with a digital camera. These captured images were employed as the inputs of the reconstruction algorithm. Fig. 9 represents the reconstruction result of the boundary of a wire in 3D space. The input stereo images are shown in Fig. 9(a) and (b). Data in these images were digitized and sampled to obtain the data points. These sampled data were used to fit NURBS representation and get the control points (see Fig. 9(c) and (d)). The reconstructed 3D boundary of the wire is shown in Fig. 9(e). Another experiment was considered in which the stereo images of a 3D vessel has been captured, which are shown in Fig. 10(a) and (c), respectively. The object of interest is the reconstruction of the surface of vessel in space. The reconstructed surface is shown in Fig. 10(e).

In the qualitative analysis, we tested the proposed algorithm for robustness. To achieve this, we added the Gaussian

noise with zero mean and varying standard deviation (r — [0.6 0.6]T) to the digitized data points in both of the views, where data were taken from the experiment no. 1 (see Fig. 4). These data points were perturbed due to the presence of noise in both of the views. Using the proposed reconstruction algorithm, the obtained reconstruction result is shown in Fig. 11(a). The projections of reconstructed curve are shown in Fig. 11(b) and (c), respectively which overlap the curve in the original image. A similar strategy was used to test the robustness in the case of surface (see Fig. 12). In this case, the Gaussian noise with zero mean and a varying standard deviation (r — [0.4 0.4]T) was added into the digitized data points in both the view planes (see Fig. 7). Fig. 12(b) shows the reconstructed surface by the proposed algorithm whereas, Fig. 12(c) represents the reconstruction result using point-based reference system Xiao and Li, 2005. Hence, the proposed algorithm performs better in the case of surfaces also where a measurable amount of noise may be present in the input stereo images.

5.2. Quantitative results

The criterion here used for computing the reconstruction error is the closest point distances (CPD's) between the reconstructed shape (in the reconstructed region) and the ground truth shape, which reflect the proximity of the two shapes. We computed the CPD's and listed these errors in various tables. Each table contains some statistics of the CPD's, i.e.,

(d) (e)

Figure 10 Surface reconstruction result of real stereo images: (a and c) Stereo images; (b and d) results of NURBS fitting with optimized control points; (e) 3D reconstructed surface.

mean, maximum and the standard deviation values under the various noise levels. Data listed in the second row of tables are the point-based reconstruction results (Xiao and Li, 2005) which is used as a reference system for the comparison study.

• The quantitative measurement of reconstruction errors in the case of experiment no. 1 (Fig. 4) is listed in Table 4.

• The quantitative measurement of surface reconstruction errors in case of experiment no. 3 (Fig. 7) is listed in Table 5.

Data in Table 4 show that the proposed reconstruction method performs better than the point-based reconstruction approach (Xiao and Li, 2005). With the support of this claim, mean, maximum and standard deviation values of the proposed (NURBS-based) reconstruction errors are less than the point-based (Xiao and Li, 2005) reconstruction results. Moreover, the proposed method achieves almost similar reconstruction results in cases when the sampling interval of image curves is not fixed. The analysis given in Table 5 concludes the same for the surfaces, but surface

reconstruction results for the proposed algorithm vary with the samplings of the image surface. However, it does not affect the quality of reconstructed surfaces. Note that in each table, e3d, e2dl and e2dr represent errors in 3D, left image plane, and right image plane respectively, and each column vector incorporates the values of mean, maximum, SD in descending order.

The quantitative analysis of the reconstruction errors in case of noisy data was conducted using the similar criterion (see Tables 6 and 7). Table 6 represents the quantitative analysis of experiment no. 1 (Fig. 4) i.e., reconstruction error in case of noisy data. Whereas Table 7 represents the quantitative analysis of the surface (experiment no. 3 (Fig. 7)) reconstruction error. Table 8 represents the reconstruction errors in the case of different curves and surfaces.

It can be seen that the reconstruction errors with the proposed algorithm are quite small in the presence of different amounts of noise. On analyzing the results, we observed that induction of stronger noise in the experiment exhibit no sign of rapid increase in 2D and 3D reconstruction error in the case of curve reconstruction as shown in Table 6. Table 7 shows the errors in 3D reconstruction with the proposed algorithm is

Figure 11 Reconstruction result of a set of 31 data points in experiment no. 1 (Fig. 4): (a) 3D reconstructed curve from corrupted stereo projections; (b and c) Corrupted stereo projections (r — [0.60.6] ) of the curve in 4(a) and the back projections of the reconstructed curve.

Table 4 Errors of curve reconstruction in experiment no. l.

Approach

rt = 0.1tin

at = 0.2t™

a, = 0.3t™

NURBS-based

Point-based

e2di =

e2dr =

?2di =

e2dr =

0.0288 0.1343 0.0281 0.0061 0.0248 0.0063 0.0237 0. 1721 0.0418

0.0367 0.4431 0.0646 0.0100 0.0744 0.0189 0.0329 0.3424 0.0954

eidi =

e2dr =

e2di =

e2dr =

0.0288 0.1345 0.0281 0.0061 0.0247 0.0063 0.0237 0.1724 0.0419

0.0349 0.1979 0.0434 0.0149 0.0789 0.0918 0.0575 0.5424 0.1058

e2di =

e2dr =

e2di =

e2dr =

0.0287 0.1346 0.0281 0.0062 0.0247 0.0063 0.0237 0.1728 0.0039

0.0502 0.3258 0.0578 0.0237 0.0819 0.0990 0.0595 0.6291 0.1198

Table 5 Errors of surface reconstruction in experiment no. 3.

Approach

at = 0.1tin

at = 0.2tin

at = 0.3tin

NURBS-based

Point-based

e2di =

e2dr =

e2di =

e2dr =

0.0873 0.3215 0.2442 0.2312 0.3315 0.2462 0.1015 0.5283 0.1810

0.2668 0.6835 0.2926 0.2838 0.6597 0.5860 0.2015 0.6932 0.2240

e2di =

e2dr =

e2di =

e2dr =

0.1382 0.5980 0.2518 0.2382 0.4823 0.1981 0.1263 0.5530 0.1999

0.2867 0.8175 0.2992 0.2715 0.6992 0.5813 0.2613 0.7930 0.2529

e2di =

e2dr =

e2di =

e2dr =

0.1471 0.6952 0.2555 0.2519 0.6028 0.2842 0.1289 0.5607 0.2017

0.2910 0.7973 0.3053 0.2763 0.7943 0.6360 0.2913 0.8192 0.2617

proportional to the increment in the amount of noise. On analyzing the results in both of the cases, we can say that it fulfills the reconstruction assumption, i.e. find the space curves and surfaces which fit the 2D image data accurately. It also shows the robustness of the proposed algorithm in the randomly posed noisy images.

Table 9 describes the significance of NURBS-based approach over the point-based reconstruction method (Xiao and Li, 2005). In other words, it supports the claim of a compressive reconstruction, i.e., the reconstruction of shape with fewer number of points (control points) in comparison

to the classical point-based approach (Xiao and Li, 2005). NURBS-based approach computes 13 control points from the given 31 data points for reconstructing 3D curve (Fig. 4), while in the case of experiment no. 2 (Fig. 6) 13 control points are reconstructed from 61 data points. In case of surface reconstruction, NURBS-based approach computes 25 control points from given 121 data points for experiment no. 3 (Fig. 7), while 80 control points are reconstructed from 400 data points to generate the surface in experiment no. 4 (Fig. 8). Hence, a significant reduction has been noticed in the number of points needed for triangulation.

Table 6 Errors of curve reconstruction with corrupted data in experiment no. 1.

Noise level «3d e2di e2dr

r = [0 . 2 0 . 2]' 0 .0489 0 0066 0 . 0316

0 2563 0 .0325 0 . 2916

0 0522 0 0067 0 0544

r = [0 . 4 0 . 4]' 0 0634 0 . 0118 0 0392

0 .2281 0 0433 0 3581

0 0556 0 0108 0 0752

r = [0 . 6 0 . 6]' 0 0790 0 . 0154 0 0372

0 2874 0 0574 0 3483

0 0670 0 0132 0 0554

r = [0 . 8 0 . 8]' 0 0824 0 . 0152 0 0598

0 2140 0 0681 0 4737

0 0550 0 0151 0 0923

r =[1 1]' 0 0752 0 0134 0 0618

0 2323 0 0458 0 5947

0 0665 0 .0119 0 .1132

Table 7 Errors of Surface Reconstruction with Corrupted Data in experiment no. 3.

Noise level «3d e2di e2dr

r = [0.1 0 .1]' 0 . 2012 0 4995 0 4943

0 3484 0 4003 0 4520

0 3328 0 0189 0.0189

r = [0 . 3 0 . 3]' 0 . 3015 0 . 5210 0 5632

0 3778 0 . 5010 0 4553

0 3623 0 0250 0 0382

r = [0 . 5 0 . 5]' 0 3385 0 5357 0 5723

0 3957 0 5582 0 4932

0 3820 0 0381 0 0463

r = [0 . 7 0 . 7]' 0 4536 0 5933 0 5962

0 5330 0 5882 0 . 5013

0 4638 0 0593 0 0629

r = [0 . 9 0 . 9]' 0 5798 0 6235 0 6532

0 6833 0 6975 0 . 6180

0 5087 0 . 0813 0 0885

Table 8 Errors of Reconstruction with Different Curves and Surfaces.

Experiment No.

r = [.1 .11-

r = [.2 . 2]'

0 .0446 0 .1343 0 .0348

0 . 3187 0 .9999 0 3234

0 .0120 0 .0755 0 . 0171

0 .4491 0 9855 0 . 3100

0 0423 0 4630 0.0931

0 4700 0 9999 0 2967

Table 9 Significance Table.

Experiment No. Number of data points used Number of reconstructed control points Resulting Figures

1 31 13 Fig. 4

2 61 13 Fig. 6

3 121 25 Fig. 7

4 400 80 Fig. 8

6. Conclusions

The presented study provides a method for reconstructing freeform curves and surfaces in 3D space from their perspective stereo views using a NURBS-based model. NURBS is used for fitting the digitized data and for acquiring the information about required shape in space. A two-step process for fitting the NURBS from the digitized data is employed. The control points, obtained in the two image-planes (by solving the linear process) are used to reconstruct the control points for required curves and surfaces in 3D space. To establish the one-to-one correspondence between control points, an extra constraint called third perspective view is used. Finally, the required shape in 3D space is reconstructed from the obtained 3D control points and their corresponding weights. Several results were presented for the evaluation of the performance of the proposed method. A comparison study was also shown in terms of various types of errors between proposed and a reference system approach.

Our experiments let out that the proposed approach is capable of reconstructing the space curves and surfaces from their perspective images more efficiently than the point-based approaches. In case of 3D curve reconstruction, proposed method performs significantly well in the presence of different sampling and noise also. Although the 3D surface reconstruction error increases with the varying sampling of image surfaces and level of noise, the proposed algorithm does not show significant difference in the quality of the results. But still there is scope for improvements. However, in terms of computational complexity, the proposed method is tremendously better as it works with few control points instead of using all the data points.

Acknowledgments

One of the authors, Deepika Saini is thankful to Ministry of Human Resources and Development (MHRD), India for the financial support to carry out this research work. Sanjeev Kumar acknowledge the support of IIT Roorkee for the grant under SRIC-FIG scheme to carry out this work.

References

An, H.M., Lee, N.C., 1996. Stereo vision based on algebraic curves. In: Presented at the 13th International Conference on Pattern Recognition, Vienna, Austria, August 25-30. Balasubramanian, R., Sukhendu, D., Swaminathan, K., 2002. Reconstruction of quadratic curves in 3-D from two or more perspective views. Math. Probl. Eng. 8, 207-219. Balasubramanian, R., Sukhendu, D., Swaminathan, K., 2003. Simulation studies for the performance analysis of the reconstruction of a line in 3-D from two arbitrary perspective views using two plane intersection method. Int. J. Comput. Math. 80, 559-571. Boot, J.C.G., 1964. Quadratic Programming, Algorithms, Anomalies

and Applications. North Holland Publishing Co., Amsterdam. Ding, M., Xiao, Y., Peng, J., Schomburg, D., Krebs, B., Wahl, M.F., 2003. 3D reconstruction of free-formed line-like objects using NURBS representation. J. Pattern Recogn. Soc. 36, 1255-1268. Farin, G., 1992. From conics to NURBS. IEEE Comput. Graph. Appl. 12, 78-86.

Faugeras, O., 1993. Three-dimensional Computer Vision: a Geometric

Viewpoint. MIT Press. Heidrich, W., Bartels, R., Labahn, G., 1996. Fitting uncertain data with NURBS. In: Proceedings of 3rd International Conference on Curves and Surfaces in Geometric Design, pp. 1-8.

Kahl, F., August, J., 2003. Multiview reconstruction of space curves. In: Proceedings of IEEE International Conference on Computer Vision, Nice, France, pp. 1017-1024.

Kahl, F., Heyden, A., 1998. Using conic correspondences in two images to estimate the epipolar geometry. In: Proceedings of the International Conference on Computer Vision, pp. 761-766.

Kanatani, K., Liu, W., 1993. 3-D interpretation of conics and orthogonality. Comput. Vis. Graph. Image Process. 58, 286-301.

Koch, R., 1995. 3-D Surface reconstruction from stereoscopic image sequences. In: Proceedings of the International Conference of Computer Vision, Cambridge, USA.

Koch, R., Pollefeys, M., Gool, V.L., 2000. Realistic surface reconstruction of 3D scenes from uncalibrated image sequences. J. Visual. Comput. Animat. 11 (3), 115-127.

Kumar, S., Kumar, M., 2012. Application of neural network in integration of shape from shading and stereo. J. King Saud Univ. -Comput. Inform. Sci. 24 (2), 129-136.

Kumar, S., Sukavanam, N., Balasubramanian, R., 2006. Reconstruction of quadratic curves in 3-D using two or more perspective views: simulation studies. In: Proceedings of the SPIE Vision Geometry XIII IS and T SPIE International Symposium on Electronic Imaging, San Jose, California, USA, 15-19 January 2006, vol. 6066, pp. 182-191.

Lavoie, P., Ionescu, D., Petriu, E., 2007. Constructing 3D virtual reality objects from 2D images of real objects using NURBS. In: IEEE International Conference on Virtual Environments, Human-Computer Interfaces, and Measurement Systems, Italy, pp. 117-124.

Luenberger, G.D., 1973. Introduction to Linear and Nonlinear Programming. Addison Wesley.

Ma, D.S., 1993. Conic-based stereo, motion estimation and pose estimation. Int. J. Comput. Vis. 10, 7-25.

Ma, W., Kruth, P.J., 1995. Parameterization of randomly measured points for least squares fitting of B-spline curves and surfaces. Comput. Aided Des. 27 (9), 663-675.

Ma, W., Kruth, P.J., 1998. NURBS curve and surface fitting for reverse engineering. Int. J. Adv. Manuf. Technol. 14, 918-927.

Peng, K., Chen, X., Zhou, D., Liu, Y., Zhai, Y., 2012. 3-D reconstruction using image sequences based on projective depth and simplified iterative closest point. Opt. Eng. 51 (2), 021110.

Piegel, L., 1991. On NURBS: a survey. IEEE Comput. Graph. Appl. 11, 55-71.

Piegel, L., Tiller, W., 1995. The NURBS Book. Springer.

Quan, L., 1996. Conic reconstruction and correspondence from two views. IEEE Trans. Pattern Anal. Mach. Intell. 18, 151-160.

Rogers, F.D., Fog, G.N., 1989. Constrained B-spline curve and surface fitting. Comput. Aided Des. 21, 641-648.

Saini, D., Kumar, S., 2014. Free-form surface reconstruction from arbitrary perspective images. In: IEEE International Advance Computing Conference (IACC), pp. 1054-1059.

Salman, M., Mansor, A., 2013. Free-form surface models generation using reverse engineering techniques - an investigation. Project Report, Universiti Sains Malaysia.

Shu, C., Roth, G., 2003. Free-form surface reconstruction from multiple images. Technical Report.

Siddiqui, M., Sclaroff, S., 2001. Surface reconstruction from multiple views using rational b-splines. Boston University Computer Science Technical Report No. 2001-009.

Sukavanam, N., Balasubramanian, R., Kumar, S., 2007. Error estimation in reconstruction of quadratic curves in 3D space. Int. J. Comput. Math. 84, 121-132.

Weng, J., Ahuja, N., Huang, S.T., 1988. Motion and structure from point correspondence: a robust algorithm for planar case with error estimation. In: Proceedings of International Conference of Pattern Recognition, ICPR, pp. 247-251.

Wong, H., Chung, R., 2003. Three-dimensional shape recovery across two views using approximate geometric constraints. Opt. Eng. 42 (3), 632-641.

Xiao, Y., Li, F.Y., 2005. Optimized stereo reconstruction of free-form space curves based on a non-uniform rational B-spline model. J. Opt. Soc. Am. 22 (9), 1746-1762. Xie, M., 1994. On 3-D reconstruction strategy: a case of conics. In:

ICPR(A), pp. 665-667. Xie, M., 1996. Thonnat M.A theory of 3-D reconstruction of heterogeneous edge primitives from two perspective views. In: ECCV, pp. 715-719. Yang, A., Huang, K., Rao, S., Hong, W., Ma, Y., 2005. Symmetry based 3-D reconstruction form perspective images. Int. J. Comput. Vis. Image Understand. 99, 210-214.

Zeng, G., Sylvain, P., Quan, L., Maxime, L, 2004. Surface reconstruction by propagating 3D stereo data in multiple 2D images. In: European Conference on Computer Vision, pp. 163-174.

Zhang, J., Feng, S., Cui, H., 2009. Optimized NURBS curve and surface fitting using simulated annealing. In: Second International Symposium on Computational Intelligence and Design.

Zhang, T., Liu, J., Tang, C., Liu, S., 2013. 3D reconstruction method of bent tube with NURBS curve. Appl. Mech. Mater. 300-301, 112-118.

Zhang, X., 2013. 3D trajectory recovery in spatial and time domains from multiple images (Ph.D. thesis).