Dr. Liang Zhao received the PhD degree in photogrammetry and remote sensing from the Institute of Remote Sensing and Geographic Information System, School of Earth and Space Science, Peking University, Beijing China, in January 2013. From 2012--2014, he was a Postdoctoral Research Fellow in the Centre for Autonomous Systems (CAS), Faculty of Engineering and Information Technology, University of Technology, Sydney (UTS), Australia. From 2014--2016, he worked as a Postdoctoral Research Associate in the Hamlyn Centre for Robotic Surgery, Department of Computing, Faculty of Engineering, Imperial College London, United Kingdom.
He is currently a Lecturer in CAS, UTS since July 2016. His current research interests include surgical robotics, simultaneous localization and mapping (SLAM), monocular SLAM (or Structure-from-Motion), aerial photogrammetry, optimization techniques in mobile robot localization and mapping and image guide robotic surgery.
He serves as reviewer for
- International Journal of Robotics Research (IJRR)
- IEEE Transactions on Intelligent Transportation Systems (T-ITS)
- IEEE Transactions on Automation Science and Engineering (T-ASE)
- ASME Journal of Dynamic Systems, Measurement and Control
- ICRA 2012, ICRA 2013, IROS 2014, ICRA 2015, IROS 2015
Can supervise: YES
- Surgical Robotics
- Simultaneous Localisation and Mapping (SLAM)
- Monocular SLAM (or Structure-from-Motion)
- Aerial Photogrammetry
- Optimisation Techniques in mobile robot localisation and mapping and image guide robotic surgery
- Advanced Robotics (49274)
- Control of Mechatronic Systems (49329)
- Sensors and Control for Mechatronic Systems (41014)
Chen, Y, Zhao, L, Lee, KMB, Yoo, C, Huang, S & Fitch, R 2020, 'Broadcast Your Weaknesses: Cooperative Active Pose-Graph SLAM for Multiple Robots', IEEE ROBOTICS AND AUTOMATION LETTERS, vol. 5, no. 2, pp. 2200-2207.View/Download from: Publisher's site
© 2013 IEEE. In this paper, we present a novel navigation framework for the Fetch robot in a large-scale environment based on submapping techniques. This indoor navigation system is divided into a submap mapping part and an on-line localization part. For the mapping part, in order to deal with large environments or multi-story buildings, a submap mapping framework fusing two-dimensional (2D) laser scan and 3D point cloud from RGBD sensor is proposed using Google Cartographer. Meanwhile, several image datasets with corresponding poses are created from the RGBD sensor. Thanks to the submap framework, the error is limited corresponding to the size of the map, thus localization accuracy will be improved. For the on-line localization, so as to switch the submaps, the on-line images from the RGBD sensor are used to match the database images using DeepLCD, a deep learning based library for loop closure. Based on the information from DeepLCD and odometry, adaptive Monte Carlo localization (AMCL) is reinitialized to finish the localization task. In order to validate the result accuracy, reflectors and a motion capture system are used to compute the absolute trajectory error (ATE) and the relative pose error (RPE) based on the Gaussian-Newton (GN) algorithm. Finally, the proposed framework is tested on the Fetch simulator and the real Fetch robot, including both submap mapping and on-line localization.
Ding, X, Wang, Y, Xiong, R, Li, D, Tang, L, Yin, H & Zhao, L 2020, 'Persistent Stereo Visual Localization on Cross-Modal Invariant Map', IEEE Transactions on Intelligent Transportation Systems, pp. 1-13.View/Download from: Publisher's site
Kong, FH, Zhao, J, Zhao, L & Huang, S 2020, 'Analysis of Minima for Geodesic and Chordal Cost for a Minimal 2-D Pose-Graph SLAM Problem', IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 323-330.View/Download from: Publisher's site
© 2016 IEEE. In this letter, we show that for a minimal 2D pose-graph SLAM problem, even in the ideal case of perfect measurements and spherical covariance, using geodesic distance (in 2D, the 'wrap function') to compare angles results in multiple suboptimal local minima. We numerically estimate regions of attraction to these local minima for some examples, give evidence to show that they are of nonzero measure, and that these regions grow in size as noise is added. In contrast, under the same assumptions, we show that the chordal distance representation of angle error has a unique minimum up to periodicity. For chordal cost, we find that initial conditions failing to converge to the global minimum are far fewer, fail because of numerical issues, and do not seem to grow with noise in our examples.
Liu, L, Zhang, T, Leighton, B, Zhao, L, Huang, S & Dissanayake, G 2019, 'Robust Global Structure From Motion Pipeline With Parallax on Manifold Bundle Adjustment and Initialization', IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 2164-2171.View/Download from: Publisher's site
© 2016 IEEE. In this letter, we present a novel global structure from motion (SfM) pipeline that is particularly effective in dealing with low-parallax scenes and camera motion collinear with the features that represent the environment structure. It is therefore particularly suitable in Urban SLAM, in which frequent road-facing motion poses many challenges to conventional SLAM algorithms. Our pipeline includes a recently explored bundle adjustment (BA) method that exploits a feature parameterization using Parallax angle between on-Manifold observation rays (PMBA). It is demonstrated that this BA stage has a consistently stable optimization configuration for features with any parallax and therefore low-parallax features can stay in reconstruction without pre-filtering. To allow practical usage of PMBA, we provide a compatible initialization stage in the SfM to initialize all camera poses simultaneously, exhibiting friendliness to collinear motion. This is achieved by simplifying PMBA into a hybrid graph problem of high connectivity yet small node set size, solved using a robust linear programming technique. Using simulations and a series of publicly available real datasets including "KITTI" and "Bundle Adjustment in the Large," we demonstrate the robustness of the position initialization stage in handling collinear motion and outlier matches, superior convergence performance of the BA stage in the presence of low-parallax features, and effectiveness of our pipeline to handle many sequential or out-of-order urban scenes.
Zhao, J, Huang, S, Zhao, L, Chen, Y & Luo, X 2019, 'Conic Feature Based Simultaneous Localization and Mapping in Open Environment via 2D Lidar', IEEE Access, vol. 7, pp. 173703-173718.View/Download from: Publisher's site
© 2013 IEEE. The conventional planar scan matching approach cannot cope well with the open environment as lacking of sufficient edges and corners. This paper presents a conic feature based simultaneous localization and mapping (SLAM) algorithm via 2D lidar which can adapt to an open environment nicely. The novelty of this work includes threefold: (1) defining a conic feature based parametrization approach; (2) developing a method to utilize feature's conic geometric information and odometry information since open environments are short of regular linear geometric features; (3) developing a factor graph based framework which can be adapted with the proposed parametrization. Simulation experiments and real environment experiments demonstrated that the proposed SLAM algorithm can get accurate and convincing results for the open environment and the map in our representation can express accurately the environment situation.
Wang, J, Song, J, Zhao, L, Huang, S & Xiong, R 2019, 'A submap joining algorithm for 3D reconstruction using an RGB-D camera based on point and plane features', Robotics and Autonomous Systems, vol. 118, pp. 93-111.View/Download from: Publisher's site
© 2019 Elsevier B.V. In standard point-based methods, the depth measurements of the point features suffer from noise, which will lead to incorrect global structure of the environment. This paper presents a submap joining based SLAM with an RGB-D camera by introducing planes as well as points as features.This work is consisted of two steps: submap building and submap joining. Several adjacent keyframes, with the corresponding small patches, visual feature points, and planes observed from these keyframes, are used to build a submap. We fuse the submaps into a global map in a sequential fashion, such that, the global structure is recovered gradually through plane feature associations and optimization. We also show that the proposed algorithm can handle plane association problem incrementally in submap level, as the plane covariance can be obtained in each submap. The use of submap significantly reduces the computational cost during the optimization process, while keeping all information about planes. The proposed method is validated using both publicly available RGB-D benchmarks and datasets collected by authors. The algorithm can produce accurate trajectories and high quality 3D models on these challenging datasets, which are difficult for existing RGB-D SLAM or SFM algorithms.
© 2018 Elsevier Ltd The main contribution of this paper is a new submap joining based approach for solving large-scale Simultaneous Localization and Mapping (SLAM) problems. Each local submap is independently built using the local information through solving a small-scale SLAM; the joining of submaps mainly involves solving linear least squares and performing nonlinear coordinate transformations. Through approximating the local submap information as the state estimate and its corresponding information matrix, judiciously selecting the submap coordinate frames, and approximating the joining of a large number of submaps by joining only two maps at a time, either sequentially or in a more efficient Divide and Conquer manner, the nonlinear optimization process involved in most of the existing submap joining approaches is avoided. Thus the proposed submap joining algorithm does not require initial guess or iterations since linear least squares problems have closed-form solutions. The proposed Linear SLAM technique is applicable to feature-based SLAM, pose graph SLAM and D-SLAM, in both two and three dimensions, and does not require any assumption on the character of the covariance matrices. Simulations and experiments are performed to evaluate the proposed Linear SLAM algorithm. Results using publicly available datasets in 2D and 3D show that Linear SLAM produces results that are very close to the best solutions that can be obtained using full nonlinear optimization algorithm started from an accurate initial guess. The C/C++ and MATLAB source codes of Linear SLAM are available on OpenSLAM.
Zhan, J, Ge, XJ, Huang, S, Zhao, L, Wong, JKW & He, SXJ 2019, 'Improvement of the inspection-repair process with building information modelling and image classification', Facilities, vol. 37, no. 7-8, pp. 395-414.View/Download from: Publisher's site
© 2019, Emerald Publishing Limited. Purpose: Automated technologies have been applied to facility management (FM) practices to address labour demands of, and time consumed by, inputting and processing manual data. Less attention has been focussed on automation of visual information, such as images, when improving timely maintenance decisions. This study aims to develop image classification algorithms to improve information flow in the inspection-repair process through building information modelling (BIM). Design/methodology/approach: To improve and automate the inspection-repair process, image classification algorithms were used to connect images with a corresponding image database in a BIM knowledge repository. Quick response (QR) code decoding and Bag of Words were chosen to classify images in the system. Graphical user interfaces (GUIs) were developed to facilitate activity collaboration and communication. A pilot case study in an inspection-repair process was applied to demonstrate the applications of this system. Findings: The system developed in this study associates the inspection-repair process with a digital three-dimensional (3D) model, GUIs, a BIM knowledge repository and image classification algorithms. By implementing the proposed application in a case study, the authors found that improvement of the inspection-repair process and automated image classification with a BIM knowledge repository (such as the one developed in this study) can enhance FM practices by increasing productivity and reducing time and costs associated with ecision-making. Originality/value: This study introduces an innovative approach that applies image classification and leverages a BIM knowledge repository to enhance the inspection-repair process in FM practice. The system designed provides automated image-classifying data from a smart phone, eliminates time required to input image data manually and improves communication and collaboration between FM personnel for maintenance i...
Song, J, Wang, J, Zhao, L, Huang, S & Dissanayake, G 2018, 'Dynamic Reconstruction of Deformable Soft-Tissue With Stereo Scope in Minimal Invasive Surgery', IEEE Robotics and Automation Letters, vol. 3, no. 1, pp. 155-162.View/Download from: Publisher's site
Song, J, Wang, J, Zhao, L, Huang, S & Dissanayake, G 2018, 'MIS-SLAM: Real-Time Large-Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computing', IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 4068-4075.View/Download from: Publisher's site
The classical absolute orientation method is capable of transforming tie points (TPs) from a local coordinate system to a global (geodetic) coordinate system. The method is based only on a unique set of similarity transformation parameters estimated by minimizing the total difference between all ground control points (GCPs) and the fitted points. Nevertheless, it often yields a transformation with poor accuracy, especially in large-scale study cases. To address this problem, this study proposes a novel absolute orientation method based on distance kernel functions, in which various sets of similarity transformation parameters instead of only one set are calculated. When estimating the similarity transformation parameters for TPs using the iterative solution of a non-linear least squares problem, we assigned larger weighting matrices for the GCPs for which the distances from the point are short. The weighting matrices can be evaluated using the distance kernel function as a function of the distances between the GCPs and the TPs. Furthermore, we used the exponential function and the Gaussian function to describe distance kernel functions in this study. To validate and verify the proposed method, six synthetic and two real datasets were tested. The accuracy was significantly improved by the proposed method when compared to the classical method, although a higher computational complexity is experienced
Vander Poorten, E, Zhao, L, Tran, P, Devreker, A, Gruijthuijsen, C, Portoles-Diez, S, Smoljkic, G, Strbac, V, Famaey, N, Reynaerts, D, Vander Sloten, J, Tibebu, A, Yu, B, Rauch, C, Bernard, F, Kassahun, Y, Metzen, JH, Giannarou, S, Lee, S, Yang, G, Mazomenos, E, Chang, P, Stoyanov, D, Kvasnytsia, M, Van Deun, J, Verhoelst, E, Sette, M, Di Iasio, A, Leo, G, Hertner, F, Scherly, D, Chelini, L, Häni, N, Seatovic, D, Rosa, B, De Praetere, H & Herijgers, P 2016, 'Cognitive AutonomouS CAtheters Operating in Dynamic Environments', Journal of Medical Robotics Research, vol. 1, no. 3.View/Download from: Publisher's site
Advances in miniaturized surgical instrumentation are key to less demanding and safer medical interventions. In cardiovascular
procedures interventionalists turn towards catheter-based interventions, treating patients considered unfit for more invasive approaches.
A positive outcome is not guaranteed. The risk for calcium dislodgement, tissue damage or even vessel rupture cannot
be eliminated when instruments are maneuvered through fragile and diseased vessels. This paper reports on the progress made in
terms of catheter design, vessel reconstruction, catheter shape modeling, surgical skill analysis, decision-making and control. These
efforts are geared towards the development of the necessary technology to autonomously steer catheters through the vasculature,
a target of the EU-funded project CASCADE (Cognitive AutonomouS CAtheters operating in Dynamic Environments). Whereas
autonomous placement of an aortic valve implant forms the ultimate and concrete goal, the technology of individual building blocks
to reach such ambitious goal is expected to be much sooner impacting and assisting interventionalists in their daily clinical practice.
Zhao, L, Giannarou, S, Lee, S-L & Yang, GZ 2016, 'SCEM+: Real-time Robust Simultaneous Catheter and Environment Modelling for Endo-vascular Navigation', IEEE Robotics and Automation Letters, vol. 1, no. 2, pp. 961-968.View/Download from: Publisher's site
Endovascular procedures are characterised by significant challenges mainly due to the complexity in catheter control and navigation. Real-time recovery of the 3-D structure of the vasculature is necessary to visualise the interaction between the catheter and its surrounding environment to facilitate catheter manipulations. State-of-the-art intraoperative vessel reconstruction approaches are increasingly relying on nonionising imaging techniques such as optical coherence tomography (OCT) and intravascular ultrasound (IVUS). To enable accurate recovery of vessel structures and to deal with sensing errors and abrupt catheter motions, this letter presents a robust and real-time vessel reconstruction scheme for endovascular navigation based on IVUS and electromagnetic (EM) tracking. It is formulated as a nonlinear optimisation problem, which considers the uncertainty in both the IVUS contour and the EM pose, as well as vessel morphology provided by preoperative data. Detailed phantom validation is performed and the results demonstrate the potential clinical value of the technique.
Sun, Y, Zhao, L, Huang, S, Yan, L & Dissanayake, G 2015, 'Line matching based on planar homography for stereo aerial images', ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING, vol. 104, pp. 1-17.View/Download from: Publisher's site
Zhao, L, Huang, S & Dissanayake, G 2015, 'Linear SFM: A Hierarchical Approach to Solving Structure-from-Motion Problems by Decoupling the Linear and Nonlinear Components', ISPRS Journal of Photogrammetry and Remote Sensing, vol. 141, pp. 275-289.View/Download from: Publisher's site
C/C++ source code
Zhao, L, Huang, S, Sun, Y, Yan, L & Dissanayake, G 2015, 'ParallaxBA: bundle adjustment using parallax angle feature parametrization', INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH, vol. 34, no. 4-5, pp. 493-516.View/Download from: Publisher's site
© 2014 Cambridge University Press. This paper presents a new monocular SLAM algorithm that uses straight lines extracted from images to represent the environment. A line is parametrized by two pairs of azimuth and elevation angles together with the two corresponding camera centres as anchors making the feature initialization relatively straightforward. There is no redundancy in the state vector as this is a minimal representation. A bundle adjustment (BA) algorithm that minimizes the reprojection error of the line features is developed for solving the monocular SLAM problem with only line features. A new map joining algorithm which can automatically optimize the relative scales of the local maps is used to combine the local maps generated using BA. Results from both simulations and experimental datasets are used to demonstrate the accuracy and consistency of the proposed BA and map joining algorithms.
Sun, Y, Zhao, L, Huang, S, Yan, L & Dissanayake, G 2014, 'L2-SIFT: SIFT feature extraction and matching for large images in large-scale aerial photogrammetry', ISPRS Journal of Photogrammetry and Remote Sensing, vol. 91, pp. 1-16.View/Download from: Publisher's site
The primary contribution of this paper is an efficient feature extraction and matching implementation for large images in large-scale aerial photogrammetry experiments. First, a Block-SIFT method is designed to overcome the memory limitation of SIFT for extracting and matching features from large photogrammetric images. For each pair of images, the original large image is split into blocks and the possible corresponding blocks in the other image are determined by pre-estimating the relative transformation between the two images. Because of the reduced memory requirement, eatures can be extracted and matched from the original images without down-sampling. Next, a red-black tree data structure is applied to create a feature relationship to reduce the search complexity when matching tie points. Meanwhile, tree key exchange and segment matching methods are proposed to match the tie points along-track and across-track. Finally, to evaluate the accuracy of the features extracted and matched from the proposed L2-SIFT algorithm, a bundle adjustment with parallax angle feature parametrization (ParallaxBA) is applied to obtain the Mean Square Error (MSE) of the feature reprojections, where the feature extraction and matching result is the only information used in the nonlinear optimisation system. Seven different experimental aerial photogrammetric datasets are used to demonstrate the efficiency and validity of the proposed algorithm. It is demonstrated that more than 33 million features can be extracted and matched from the Taian dataset with 737 images within 21h using the L2-SIFT algorithm. In addition, the ParallaxBA involving more than 2.7 million features and 6 million image points can easily converge to an MSE of 0.03874. The C/C++ source code for the proposed algorithm is available at http://services.eng.uts.edu.au/~sdhuang/research.htm.
Chen, Y, Huang, S, Fitch, R, Zhao, L, Yu, H & Yang, D 2019, 'On-line 3D active pose-graph SLAM based on key poses using graph topology and sub-maps', Proceedings - IEEE International Conference on Robotics and Automation, pp. 169-175.View/Download from: Publisher's site
© 2019 IEEE. In this paper, we present an on-line active pose-graph simultaneous localization and mapping (SLAM) frame-work for robots in three-dimensional (3D) environments using graph topology and sub-maps. This framework aims to find the best trajectory for loop-closure by re-visiting old poses based on the T-optimality and D-optimality metrics of the Fisher information matrix (FIM) in pose-graph SLAM. In order to reduce computational complexity, graph topologies are introduced, including weighted node degree (T-optimality metric) and weighted tree-connectivity (D-optimality metric), to choose a candidate trajectory and several key poses. With the help of the key poses, a sampling-based path planning method and a continuous-time trajectory optimization method are combined hierarchically and applied in the whole framework. So as to further improve the real-time capability of the method, the sub-map joining method is used in the estimation and planning process for large-scale active SLAM problems. In simulations and experiments, we validate our approach by comparing against existing methods, and we demonstrate the on-line planning part using a quad-rotor unmanned aerial vehicle (UAV).
Zhu, H, Leighton, B, Chen, Y, Ke, X, Liu, S & Zhao, L 2019, 'Indoor Navigation System Using the Fetch Robot', Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 686-696.View/Download from: Publisher's site
© 2019, Springer Nature Switzerland AG. In this paper, we present a navigation system, including off-line mapping and on-line localization, for the Fetch robot in an indoor environment using Cartographer. This framework aims to build a practical, robust, and accurate Robot Operating System (ROS) package for the Fetch robot. Firstly, using Cartographer and the fusion of data from a laser scan and RGB-D camera, a two-dimensional (2D) off-line map is built. Then, the Adaptive Monte Carlo Localization (AMCL) ROS package is used to perform on-line localization. We use a simulation to validate this method of mapping and localization, then demonstrate our method live on the Fetch robot. A video about the simulation and experiment is shown in https://youtu.be/oOvxTOowe34.
Liu, L, Zhang, T, Liu, Y, Leighton, B, Zhao, L, Huang, S & Dissanayake, G 2018, 'Parallax Bundle Adjustment on Manifold with Improved Global Initialization', Springer Proceedings in Advanced Robotics (SPAR), International Workshop on the Algorithmic Foundations of Robotics, Springer, Mérida, México.
In this paper we present a novel extension to the parallax feature based
bundle adjustment (BA). We take parallax BA into a manifold form (PMBA)
along with an observation-ray based objective function. This formulation faithfully mimics the projective nature in a camera’s image formation, resulting in a stable optimization configuration robust to low-parallax features. Hence it allows use of fast Dogleg optimization algorithm, instead of the usual Levenberg Marquardt. This is particularly useful in urban SLAM in which diverse outdoor environments and collinear motion modes are prevalent. Capitalizing on these properties, we propose a global initialization scheme in which PMBA is simplified into a pose-graph problem. We show that near-optimal solution can be achieved under low-noise conditions. With simulation and a series of challenging
publicly available real datasets, we demonstrate PMBA’s superior convergence performance in comparison to other BA methods. We also demonstrate, with the “Bundle Adjustment in the Large” datasets, that our global initialization process successfully bootstrap the full BA in mapping many sequential or out-of-order urban scenes.
Zhao, J, Huang, S & Zhao, L 2018, 'Constrained Gaussian Mixture Models Based Scan Matching Method', ACRA 2018 Website Proceedings, Australasian Conference on Robotics and Automation, ARAA, New Zealand, pp. 1-8.
This paper presents a Gaussian mixture model
(GMM) based robust scan matching method
which implements GMM to represent 2D scan
points and improves the accuracy of scan
matching. The proposed method transfers each
new scan to GMM first, exploiting the covariance of every GMM component to represent
scan points. Compared with the conventional
GMM based method of scan matching, our
technique implements GMM similarity comparison to evaluate the overlaps between scans.
In order to get rid of the poor convergence
due to the inaccurate initial value given to
the iteration process, we proposed a geometryconstraint-based GMM similarity calculation
method, which is one contribution of this paper. Another contribution is we propose a dynamic scale factor making the cost function
more adapted to different initial value. Experiments on simulated data are employed and the
results indicate that our method is able to enlarge the valid range of initial value and accumulate small errors after sequential matchings.
Liu, L, Wang, Y, Zhao, L & Huang, S 2017, 'Evaluation of Different SLAM Algorithms using Google Tangle Data', 2017 12th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE Conference on Industrial Electronics and Applications, IEEE, Siem Reap, Cambodia..View/Download from: Publisher's site
In this paper, we evaluate three state-of-the-art Simultaneous Localization and Mapping (SLAM) methods using data extracted from a state-of-the-art device for indoor navigation - the Google Tango tablet. The SLAM algorithms we investigated include Preintegration Visual Inertial Navigation System (VINS), ParallaxBA and ORB-SLAM. We first describe the detailed process of obtaining synchronized IMU and image data from the Google Tango device, then we present some of the SLAM results obtained using the three different SLAM algorithms, all with the datasets collected from Tango. These SLAM results are compared with that obtained from Tango's inbuilt motion tracking system. The advantages and failure modes of the different SLAM algorithms are analysed and illustrated thereafter. The evaluation results presented in this paper are expected to provide some guidance on further development of more robust SLAM algorithms for robotic applications.
Leighton, B, Zhao, L, Huang, S & Dissanayake, G 2017, 'Extending Parallax Parameterised Bundle Adjustment to Stereo', Australasian Conference on Robotics and Automation 2017, Australasian Conference on Robotics and Automation, ARAA, University of Technology, Sydney, pp. 1-9.
The main contribution of this paper is the extension
of the ParllaxBA algorithm proposed
by [Zhao et al., 2015] into stereo. Simulated
and experimental datasets are used to evaluate
Cartesian and parallax angle parameterisation
for stereo bundle adjustment. It is demonstrated
that, like monocular ParallaxBA, under
normal conditions the two algorithms perform
similarly. However, when the parallax angle
of landmarks is low, parallax parameterisation
can converge to a lower cost and in less
time than the traditional Cartesian parameterisation
Song, J, wang, J, Zhao, L, Huang, S & Dissanayake, G 2017, 'Deformable Soft-tissue Reconstruction using Stereo Scope for Minimal Invasive Surgery', CARS 2017 -- Computer Assisted Radiology and Surgery, 31st International Congress and Exhibition, CARS 2017 -- Computer Assisted Radiology and Surgery, 31st International Congress and Exhibition.
song, J, wang, J, Zhao, L, huang, S & Dissanayake, G 2017, 'Robust Shape Recovery of Deformable Soft-tissue Based on Information from Stereo Scope for Minimal Invasive Surgery', Hamlyn Symposium on Medical Robotics (HSMR 2017), Hamlyn Symposium on Medical Robotics, Kensington, England.
Overcoming small field of view of scopes is an important challenge in minimal invasive surgery (MIS). Efforts have been devoted in 3D soft-tissue construction and camera localization [1-2]. This paper proposes a
robust strategy for simultaneous camera localization and dense reconstruction of deformable surfaces. The robustness is achieved by: (1) using a sequence of images collected from a stereoscope by considering
uncertainty map; (2) filtering images with low intensity; (3) filtering depth by normals. Our approach greatly reduces depth estimation parameter adjustment efforts while still generates good results and preserves
topological details. Experiments reveal that the proposed approach is convenient for dynamically rebuild and visualize the latest shape of soft-tissue to mitigate unnecessary tissue damages in minimal
Wang, J, Song, J, Zhao, L & Huang, S 2017, 'A Submap Joining Based RGB-D SLAM Algorithm using Planes as Features', 11th Conference on Field and Service Robotics (FSR 2017), 11th Conference on Field and Service Robotics (FSR 2017), Springer, Zurich, Switzerland.
Chen, S, Doan, VH & Zhao, L 2017, 'Heart simulator: A periodic pump to simulate the cardiac motion in an aortic test-rig', Australasian Conference on Robotics and Automation, ACRA, Australasian Conference on Robotics and Automation, ARAA, Sydney, Australia, pp. 100-106.
A periodic pump that simulates blood ejection from human heart to aorta is the core element for building an aortic robotics test-rig. This paper is to describe the design of such a prototype human heart simulator and its performance under different working status, such as simulating the physiological states of a healthy adult and/or a child in sleep, relax and physical exercise. By balancing the cost and performance, this prototype has these specifications: (1) Using ordinary plumbing components and water to simulate the cardiac motion and blood flow. (2) Simulating the volume change of human heart chamber by controlling movement of a mechanical piston. (3) Performing a friendly user interface and delicate control via a MCU system with high reliability. (4) Simulated physiological output parameters such as volume per stroke, heart beat rate and waveform can be easily adjusted and monitored in real-time.
Wang, J, Huang, S, Zhao, L, Ge, X, He, X, Zhang, C & Wang, X 2017, 'High Quality 3D Reconstruction of Indoor Environments using RGB-D Sensors', Proceedings of the 2017 12th IEEE Conference on Industrial Electronics and Applications (ICIEA), IEEE 12th Conference on Industrial Electronics and Applications (ICIEA), IEEE, Siem Reap, Cambodia, pp. 1736-1741.View/Download from: Publisher's site
High-quality 3D reconstruction of large-scale indoor scene is the key to combine Simultaneous Localization And Mapping (SLAM) with other applications, such as building inspection and construction monitoring. However, the requirement of global consistency brings challenges to both localization and mapping. In particular, significant localization and mapping error can happen when standard SLAM techniques are used when dealing with the area of featureless walls and roofs. This paper proposed a novel framework aiming to reconstruct a high-quality, globally consistent 3D model for indoor environments using only a RGB-D sensor. We first introduce the sparse and dense feature constraints in the local bundle adjustment. Then, the planar constraints are incorporated in the global bundle adjustment. We fuse the point clouds in a truncated signed distance function volume, from which the high quality mesh can be extracted. Our framework leads to a comprehensive 3D scanning solution for indoor scene, enabling high-quality results and potential applications in building information system. The video of 3D models reconstructed by the method proposed in this paper is available at https://youtu.be/DWMP4YfeNeY.
Song, JW, Wang, J, Zhao, L, Huang, S & Dissanayake, G 2016, '3D Shape Recovery of Deformable Soft-tissue with Computed Tomography and Depth Scan', Website proceedings of the Australasian Conference on Robotics and Automation 2016, Australasian Conference on Robotics and Automation, ARAA, Queensland, Australia, pp. 1-9.
Knowing the tissue environment accurately is
very important in minimal invasive surgery
(MIS). While, as the soft-tissues is deformable,
reconstruction of the soft-tissues environment
is challenging. This paper proposes a new
framework for recovering the deformation of
the soft-tissues by using a single depth sensor.
This framework makes use of the morphology
information of the soft-tissues from Xray
computed tomography, and deforms it by
the embedded deformation method. Here, the
key is to build a distance field function of the
scan from the depth sensor, which can be used
to perform accurate model-to-scan deformation
together with robust non-rigid shape registration
in the same go. Simulations show that
soft-tissue shape in the previous step can be ef-
ficiently deformed to fit the partially observed
scan in the current step by using the proposed
method. And the results from the simulated
sequential deformation of three different softtissues
demonstrate the potential clinical value
Zhao, L, Giannarou, S, Lee, S-L & Yang, GZ 2016, 'Registration-free Simultaneous Catheter and Environment Modelling', Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 (LNCS), Medical Image Computing and Computer-Assisted Intervention, Springer, Athens, Greece, pp. 525-533.View/Download from: Publisher's site
Endovascular procedures are challenging to perform due to the complexity and difficulty in catheter manipulation. The simultaneous recovery of the 3D structure of the vasculature and the catheter position and orientation intra-operatively is necessary in catheter control and navigation. State-of-art Simultaneous Catheter and Environment Modelling provides robust and real-time 3D vessel reconstruction based on real-time intravascular ultrasound (IVUS) imaging and electromagnetic (EM) sensing, but still relies on accurate registration between EM and pre-operative data. In this paper, a registration-free vessel reconstruction method is proposed for endovascular navigation. In the optimisation framework, the EM-CT registration is estimated and updated intra-operatively together with the 3D vessel reconstruction from IVUS, EM and pre-operative data, and thus does not require explicit registration. The proposed algorithm can also deal with global (patient) motion and periodic deformation caused by cardiac motion. Phantom and in-vivo experiments validate the accuracy of the algorithm and the results demonstrate the potential clinical value of the technique.
Zhao, L, Giannarou, S, Lee, S-L, Merrifield, R & Yang, GZ 2016, 'Intra-operative Simultaneous Catheter and Environment Modelling for Endovascular Navigation Based on Intravascular Ultrasound, Electromagnetic Tracking and Pre-operative Data', Proceedings of The Hamlyn Symposium on Medical Robotics, The Hamlyn Symposium on Medical Robotics, Imperial College London and the Royal Geographical Society, London, UK, pp. 76-77.
Cardiovascular diseases (CVD) form the single most
common cause of death. Catheter procedures are among
the most common surgical interventions used to treat
CVD. Due to their minimal access trauma, these
procedures extend the range of patients able to receive
interventional CVD treatment to age groups dominated
by co-morbidity and unacceptable risks for open surgery
. The downside associated with minimising access
incisions lies at the increased complexity and difficult
manipulation of the instruments and anatomical targets,
which is mainly caused by the loss of direct access to
the anatomy and the poor visualisation of the surgical
site. The current clinical approaches to endovascular
procedures mainly rely on 2D guidance based on X-ray
fluoroscopy, which uses ionising radiation and
dangerous contrast agents .
In this paper, a Simultaneous Catheter and Environment
Modelling (SCEM) method is presented for
endovascular navigation based on intravascular
ultrasound (IVUS) imaging, electromagnetic (EM)
sensing as well as the vessel structure information
provided from the pre-operative CT/MR imaging (see
Fig. 1). Thus, radiation dose and contrast agents are
avoided. The proposed SCEM intra-operatively recovers
the 3D structure of the vasculature together with the
pose of the catheter tip, which the knowledge of the
interaction between the catheter and its surroundings
can be provided. The corresponding uncertainties of
both vessel reconstruction and catheter pose can also be
computed which is necessary for autonomous robotic
catheter navigation. Experimental results using three
different phantoms, with different catheter motions and
cardiac motions simulated by using a periodic pump
demonstrated the accuracy of the vessel reconstruction
and the potential clinical value of the proposed SCEM
Zhao, L, Huang, S & Dissanayake, G 2014, 'Linear MonoSLAM: A Linear Approach to Large-Scale Monocular SLAM Problems', 2014 IEEE International Conference on Robotics & Automation (ICRA), IEEE International Conference on Robotics and Automation, IEEE, Hong Kong, China, pp. 1517-1523.View/Download from: Publisher's site
This paper presents a linear approach for solving monocular simultaneous localization and mapping (SLAM) problems. The algorithm rst builds a sequence of small initial submaps and then joins these submaps together in a divideand-conquer (D&C) manner. Each of the initial submap is built using three monocular images by bundle adjustment (BA), which is a simple nonlinear optimization problem. Each step in the D&C submap joining is solved by a linear least squares together with a coordinate and scale transformation. Since the only nonlinear part is in the building of the initial submaps, the algorithm makes it possible to solve large-scale monocular SLAM while avoiding issues associated with initialization, iteration, and local minima that are present in most of the nonlinear optimization based algorithms currently used for large-scale monocular SLAM. Experimental results based on publically available datasets are used to demonstrate that the proposed algorithms yields solutions that are very close to those obtained using global BA starting from good initial guess.
Zhao, L, Huang, S & Dissanayake, G 2013, 'Linear SLAM: A Linear Solution to the Feature-based and Pose Graph SLAM based on Submap Joining', 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Tokyo, Japan, pp. 24-30.View/Download from: Publisher's site
This paper presents a strategy for large-scale SLAM through solving a sequence of linear least squares problems. The algorithm is based on submap joining where submaps are built using any existing SLAM technique. It is demonstrated that if submaps coordinate frames are judiciously selected, the least squares objective function for joining two submaps becomes a quadratic function of the state vector. Therefore, a linear solution to large-scale SLAM that requires joining a number of local submaps either sequentially or in a more efficient Divide and Conquer manner, can be obtained. The proposed Linear SLAM technique is applicable to both feature-based and pose graph SLAM, in two and three dimensions, and does not require any assumption on the character of the covariance matrices or an initial guess of the state vector. Although this algorithm is an approximation to the optimal full nonlinear least squares SLAM, simulations and experiments using publicly available datasets in 2D and 3D show that Linear SLAM produces results that are very close to the best solutions that can be obtained using full nonlinear optimization started from an accurate initial value. The C/C++ and MATLAB source codes for the proposed algorithm are available on OpenSLAM.
Ahmad, A, Zhao, L, Huang, S & Dissanayake, G 2012, 'Convergence comparison of least squares based bearing-only SLAM algorithms using different landmark parametrizations', 2012 International Conference on Control, Automation, Robotics & Vision (ICARCV ), International Conference on Control, Automation, Robotics and Vision, IEEE, Guangzhou, China, pp. 1006-1011.View/Download from: Publisher's site
This paper compares the convergence of least squares based 2D bearing-only SLAM algorithms using different landmark parametrizations. It is shown that the requirement on the accuracy of the initial value vary significantly when using different landmark parametrizations. Especially, for small scale bearing-only SLAM problems, the region of attraction of the global minimum for Gauss-Newton iteration based bearing-only SLAM algorithm using parallax angle landmark parametrization is significantly larger as compared with those of bearing-only SLAM algorithms using other landmark parametrizations.
Himstedt, M, Alempijevic, A, Zhao, L, Huang, S & Boehme, H 2012, 'Towards robust vision-based self-localization of vehicles in dense urban environments', 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Algarve, Portugal, pp. 3152-3157.View/Download from: Publisher's site
Self-localization of ground vehicles in densely populated urban environments poses a significant challenge. The presence of tall buildings in close proximity to traversable areas limits the use of GPS-based positioning techniques in such environments. This paper presents an approach to global localization on a hybrid metric-topological map using a monocular camera and wheel odometry. The global topology is built upon spatially separated reference places represented by local image features. In contrast to other approaches we employ a feature selection scheme ensuring a more discriminative representation of reference places while simultaneously rejecting a multitude of features caused by dynamic objects. Through fusion with additional local cues the reference places are assigned discrete map positions allowing metric localization within the map. The self-localization is carried out by associating observed visual features with those stored for each reference place. Comprehensive experiments in a dense urban environment covering a time difference of about 9 months are carried out. This demonstrates the robustness of our approach in environments subjected to high dynamic and environmental changes.
Hu, G, Huang, S, Zhao, L, Alempijevic, A & Dissanayake, G 2012, 'A Robust RGB-D SLAM algorithm', 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Algarve, Portugal, pp. 1174-1179.View/Download from: Publisher's site
Recently RGB-D sensors have become very popular in the area of Simultaneous Localisation and Mapping (SLAM). The major advantage of these sensors is that they provide a rich source of 3D information at relatively low cost. Unfortunately, these sensors in their current forms only have a range accuracy of up to 4 metres. Many techniques which perform SLAM using RGB-D cameras rely heavily on the depth and are restrained to office type and geometrically structured environments. In this paper, a switching based algorithm is proposed to heuristically choose between RGB-BA and RGBD-BA based local maps building. Furthermore, a low cost and consistent optimisation approach is used to join these maps. Thus the potential of both RGB and depth image information are exploited to perform robust SLAM in more general indoor cases. Validation of the proposed algorithm is performed by mapping a large scale indoor scene where traditional RGB-D mapping techniques are not possible.
Zhao, L, Huang, S, Yan, L & Dissanayake, G 2011, 'Parallax angle parametrization for monocular SLAM', 2011 IEEE International Conference on Robotics and Automation (ICRA), IEEE International Conference on Robotics and Automation, IEEE, Shanghai, China, pp. 3117-3124.View/Download from: Publisher's site
This paper presents a new unified feature parametrization approach for monocular SLAM. The parametrization is based on the parallax angle and can reliably represent both nearby and distant features, as well as features in the direction of camera motion and features observed only once. A new bundle adjustment (BA) algorithm using the proposed parallax angle parametrization is developed and shown to be more reliable as compared with existing BA algorithms that use Euclidean XYZ or inverse depth parametrizations. A new map joining algorithm that allows combining a sequence of local maps generated using BA with the proposed parametrization, that avoids the large computational cost of a global BA, and can automatically optimize the relative scales of the local maps without any loss of information, is also presented. Extensive simulations and a publicly available large-scale real dataset with centimeter accuracy ground truth are used to demonstrate the accuracy and consistency of the BA and map joining algorithms using the new parametrization. Especially, since the relative scales are optimized automatically in the proposed BA and map joining algorithms, there is no need to compute any relative scales even for a loop more than 1km.
Zhao, L, Huang, S, Yan, L, Wang, J, Hu, G & Dissanayake, G 2010, 'Large-Scale Monocular SLAM by Local Bundle Adjustment and Map Joining', 2010 International Conference on Control, Automation, Robotics and Vision (ICARCV), IEEE, Singapore, pp. 431-436.View/Download from: Publisher's site
This paper first demonstrates an interesting property of bundle adjustment (BA), âscale drift correctionâ. Here âscale drift correctionâ means that BA can converge to the correct solution (up to a scale) even if the initial values of the camera pose translations and point feature positions are calculated using very different scale factors. This property together with other properties of BA makes it the best approach for monocular Simultaneous Localization and Mapping (SLAM), without considering the computational complexity. This naturally leads to the idea of using local BA and map joining to solve large-scale monocular SLAM problem, which is proposed in this paper. The local maps are built through Scale-Invariant Feature Transform (SIFT) for feature detection and matching, random sample consensus (RANSAC) paradigm at different levels for robust outlier removal, and BA for optimization. To reduce the computational cost of the large-scale map building, the features in each local map are judiciously selected and then the local maps are combined using a recently developed 3D map joining algorithm. The proposed large-scale monocular SLAM algorithm is evaluated using a publicly available dataset with centimeter-level ground truth.
C/C++ source code
Sun, Y & Zhao, L 2015, 'Line Matching based on Planar Homography for Aerial Photogrammetric Images C/C++ source code'.
C/C++ source code
MATLAB and C/C++ source code
C/C++ source code
Sun, Y & Zhao, L 2013, 'L2-SIFT: SIFT Feature Extraction and Matching for Large Images in Large-scale Aerial Photogrammetry C/C++ source code'.
C/C++ source code
- Imperial College London
- University of Leeds
- Peking University
- Zhejiang University
- Tianjin University
- Northeast University