The point cloud reconstruction method used here is based on spring embedders, which perform iterative constraint optimizations. Disadvantages of spring embedder algorithms are their slow convergence. As mentioned before, force-directed methods may also be evaluated by solving differential equations. Future work would be to investigate more efficient implementations of the reconstruction algorithm. This is, how-ever, a step that is best postponed until experiments with new heurstics or constraints have been completed, since the spring embedder approach is easily adapted to accomodate for new data. Furthermore, the recon-struction method has not been implemented on a handheld device yet (see Section A for implementation
64
details), which would be another direction to take.
An interesting question would be whether the camera translation and rotation can be optimized sequentially.
When the camera rotation estimates of the sensor fusion approach proved to be prone to error, we performed an experiment in which forces were calculated for both the camera rotation and translation, so that both were updated either simultaneously or alternately. This experiment often yielded a reprojection error that was far worse than the reprojection error obtained when only the camera translation was adapted. The problem lies in the translation-rotation ambiguity, meaning that translation and rotation are sometimes difficult to distinguish. For example, suppose that all reprojections lie to the left (as seen through a virtual camera) of their corresponding interest points. One would have to analyse the reprojection pattern to determine whether the camera needs to be translated or rotated to the left. Especially when reprojections may lie at any side of their corresponding interest point, an advanced analysis of the reprojection pattern may be necessary to determine the motion that is required to optimize the reprojection error.
7.3 Meshing
Minor improvements could be made to the meshing step. As noted, the Ball Pivoting algorithm does not require the surface normals of the 3D points to be known. If the surface normals are not known, the al-gorithm estimates the surface normals of the mesh triangles, which could result in the wrong side being marked as the front face of the triangle. Since, after the reconstruction, the poses of the virtual cameras are reconstructed, each point has two or more projection lines onto the image plane of these virtual cameras, from which surface normals can be estimated that are likely to be a better estimate than those determined by the Ball Pivoting algorithm.
Furthermore, after the reconstruction the mesh might be further refined by obtaining additional informa-tion from the captured images. One way would be to fix the camera poses and 3D points and to extract new interest points from the captured images. This yields new 3D points that are to be reconstructed, this time with fixed camera poses. Many new interest points may be obtained by redoing the interest point detection and matching without the strict requirements (symmetrical matches, ratio testing or use of the fundamental matrix): Poorly matched interest points of this new set do not influence the existing camera poses or the existing point cloud coordinates and can be discarded based on their poor reprojection error after this second reconstruction has completed. The expectation is that this yields a more detailed mesh. Another area of re-search might be to apply a contour finding algorithm to the captured images and to use the detected contours to further refine the mesh.
Lastly, an extension would be to obtain textures from the captured images and to apply these to the mesh, a minor change that will nevertheless contribute greatly to the visual appeal of the obtained reconstruction.
Implementation Specifics
The resulting implementation consists of two applications:
• An Android application that captures images and estimates the camera’s viewing angle. This appli-cation was developed and tested on a Samsung Galaxy SII with a 1.2 GHz CPU and 1GB of RAM, running the Android 4.0.3 operating system.
• An application that processes the captured images, extracts data and performs the 3D point cloud reconstruction. This application was developed in C++ under Linux and tested on a Thinkpad T42 with an Intel Pentium M 1.7 GHz CPU and 1GB of RAM.
Both applications have been developed with the use of OpenCV [34], an open source computer vision library, available online athttp://code.opencv.org/projects/opencv.
66
[1] S. Winkelbach, S. Molkenstruck, and F. Wahl, “Low-Cost Laser Range Scanner and Fast Surface Regis-tration Approach,” in Pattern Recognition, Vol. 4174 of Lecture Notes in Computer Science, K. Franke, K.-R. MÃijller, B. Nickolay, and R. SchÃd’fer, eds., (Springer Berlin Heidelberg, 2006), pp. 718–728.
[2] P. A. Eades, “A heuristic for graph drawing.,” In Congressus Numerantium, 42, 149–160 (1984).
[3] T. Kamada and S. Kawai, “An algorithm for drawing general undirected graphs,” Inf. Process. Lett. 31, 7–15 (1989).
[4] T. M. J. Fruchterman and E. M. Reingold, “Graph drawing by force-directed placement,” Softw: Pract.
Exper. 21, 1129–1164 (1991).
[5] D. C. Brown, “Decentering distortion of lenses,” Photogrammetric Engineering and Remote Sensing 32, 444–462 (1966).
[6] D. C. Brown, “Close-range camera calibration,” Photogrammetric Engineering and Remote Sensing 37, 855–866 (1971).
[7] E. Hemayed, “A survey of camera self-calibration,” In Proceedings of the IEEE Conference on Ad-vanced Video and Signal Based Surveillance, 2003, AVSS’03 pp. 351–357 (2003).
[8] Z. Zhang, “A Flexible New Technique for Camera Calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1330–1334 (2000).
[9] K. Levenberg, “A method for the solution of certain non-linear problems in least squares,” Quart. J.
Appl. Maths. II, 164–168 (1944).
[10] G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library (O’Reilly, Cambridge, MA, 2008).
[11] R. Luo, Y. C. Chou, and O. Chen, “Multisensor Fusion and Integration: Algorithms, Applications, and Future Research Directions,” In Mechatronics and Automation, 2007. ICMA 2007. International Conference on, ICMA 2007 pp. 1986 –1991 (2007).
[12] W. Higgins, “A Comparison of Complementary and Kalman Filtering,” Aerospace and Electronic Sys-tems, IEEE Transactions on AES-11, 321 –325 (1975).
[13] S. Colton, “The Balancing Filter,”, 2007,http://www.chiefdelphi.com/media/papers/2010.
[14] P. Lawitzki, Master’s thesis, Stuttgart Media University, 2012, “Application of Dynamic Binaural Sig-nals in Acoustic Games”.
67
[15] D. Hardy, Master’s thesis, Eindhoven University of Technology, 2011, “Augmented Reality for Land-scape Development on Mobile Devices”.
[16] A. Somers, Master’s thesis, Eindhoven University of Technology, 2012, “Markerless GPU Accelerated Augmented Reality on Android Devices”.
[17] Ava group of the University of Cordoba, Spain, “ArUco: Augmented Reality library,”, 2012.
[18] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-Up Robust Features (SURF),” Comput. Vis.
Image Underst. 110, 346–359 (2008).
[19] T. Tuytelaars and K. Mikolajczyk, “Local invariant feature detectors: a survey,” Found. Trends. Com-put. Graph. Vis. 3, 177–280 (2008).
[20] M. Brown and D. Lowe, “Invariant Features from Interest Point Groups,” In In British Machine Vision Conference, pp. 656–665 (2002).
[21] D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int. J. Comput. Vision 60, 91–110 (2004).
[22] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with appli-cations to image analysis and automated cartography,” Commun. ACM 24, 381–395 (1981).
[23] N. Snavely, S. M. Seitz, and R. Szeliski, “Modeling the World from Internet Photo Collections,” Int. J.
Comput. Vision 80, 189–210 (2008).
[24] S. Bougnoux, “From projective to Euclidean space under any practical situation, a criticism of self-calibration,” In Computer Vision, 1998. Sixth International Conference on, pp. 790 –796 (1998).
[25] S. G. Kobourov, “Spring Embedders and Force Directed Graph Drawing Algorithms,” In Handbook of Graph Drawing and Visualization, R. Tamassia, ed., (2012).
[26] G. Di Battista, P. Eades, R. Tamassia, and I. G. Tollis, “Algorithms for drawing graphs: an annotated bibliography,” Comput. Geom. Theory Appl. 4, 235–282 (1994).
[27] P. Cignoni, M. Corsini, and G. Ranzuglia, “MeshLab: an Open-Source 3D Mesh Processing System,”
ERCIM News pp. 45–46 (2008), available online athttp://meshlab.sourceforge.net/.
[28] F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, G. Taubin, and S. Member, “The Ball-Pivoting Algorithm for Surface Reconstruction,” IEEE Transactions on Visualization and Computer Graphics 5, 349–359 (1999).
[29] M. Kazhdan, M. Bolitho, and H. Hoppe, “Poisson surface reconstruction,” In Proceedings of the fourth Eurographics symposium on Geometry processing, SGP ’06 pp. 61–70 (Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 2006).
[30] M. Kazhdan, T. Funkhouser, and S. Rusinkiewicz, “Shape matching and anisotropy,” In ACM SIG-GRAPH 2004 Papers, SIGSIG-GRAPH ’04 pp. 623–629 (ACM, New York, NY, USA, 2004).
[31] B. Zeisl, P. F. Georgel, F. Schweiger, E. Steinbach, and N. Navab, “Estimation of Location Uncertainty for Scale Invariant Feature Points,” In Proceedings of the British Machine Vision Conference, pp.
57.1–57.12 (BMVA Press, 2009), doi:10.5244/C.23.57.
[32] R. M. Steele and C. Jaynes, “Feature Uncertainty Arising from Covariant Image Noise,” In Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR
’05 pp. 1063–1070 (IEEE Computer Society, Washington, DC, USA, 2005).
[33] “Computer Graphics Lab., The Stanford 3D Scanning Repository,”, 2008.
[34] “The OpenCV Library,”, 2012, available online athttp://code.opencv.org/projects/opencv.
Camera
calibration, 9 extrinsics, 5 intrinsics, 5 pose, 6
resectioning, 9 epipolar line, 27 feature detector, 21 focal length, 5
fundamental matrix, 27 Hessian matrix, 21
mean absolute deviation, 62 OpenCV, 66
optimal fit, 37 principal point, 5 radial distortion, 5 RANSAC, 27
reprojection error, 33 sub-pixel accuracy, 24 tangential distortion, 5
translation-rotation ambiguity, 65 virtual camera, 31
70