Works and Projects

Core Developer · PICO MR cloud-based mapping and localization — Visual Positioning Service (VPS).

Project Owner · High-performance Bundle Adjustment for large-scale scenes — Ceres CUDA, FastBA, MegBA.

Core Developer · Real-time sharing and localization of anchor maps across multiple devices, enabling multi-player MR games.

Project Owner · Vision-based, low-cost mapping and localization powering MR Spatial Anchors on PICO headsets.

Project Owner · Line-feature based VIO designed to improve re-localization performance on MR headsets.

Project Owner · Fuse cloud-based high-precision localization results into the on-device 6DoF tracking loop.

Project Owner · Use IMU data to compensate for the rolling-shutter “jelly” effect in RGB cameras under fast motion.

Core Developer · Multi-camera and lidar extrinsic calibration with a re-engineered workflow.

Project Owner · End-to-end OCR toolbox covering detection, recognition, post-processing, and TensorRT acceleration.

Project Owner · Camera-based semantic mapping for parking environments using line and marker features.

Project Owner · AGV-level multi-sensor fusion module for static and dynamic obstacle state estimation.

Core Developer · Evaluation toolboxes covering classification, keypoint detection, keypoint regression, and image-based 3D-box tasks.

Project Owner · Vehicle and cyclist 3D bounding box detection for the auto-driving perception stack.

Project Owner · Camera IPM-based parking space detection that emits 3D coordinates for downstream parking planning.

source code
demo 1 / demo 2

1. Within this project, we implemented the Visual Inertial Odometry to estimate the states of a Quadrotor, including its global position [x, y, z], pose [roll, pitch, yaw] and linear velocity with respected to the world [vx, vy, vz].
2. We achieved both simulation on Linux/ROS and lab environment test on a real Quadrotor tasks.

Implemented two of important robotics required algorithms.

1. Polynomial Trajectory Planning and Generation: design and generate an optimal polynomial smooth trajectory for the Quad rotor given a specified 3D Map. source code
2. Hand-handled Camera Calibration: aimed to estimate the intrinsic (K) and extrinsic (R, t) matrices for the camera using a planer world space(e.g. checkerboard) and multiple views, and optimized all variables using the reprojection errors. source code

source code
Built a deep network to achieve the 3D-MNIST data classification, 3D bounding box estimation and cube plane depth estimation.

1. 3D Data (3D MNIST) Pre-Processing, including the Voxel generation and 2D multiple views generation.
2. Deep network construction which can estimate class label, depth map and 2D bbox in one single stage.
3. 3D bounding box estimation.

Mastered the algorithm and implementation of most generative models, including AE, VAE and multiple GANs.

1. Implemented AE(Auto-Encoder) and VAE(Variational AutoEncoder) on people face data, e.g. cufs human faces. source code
2. Researched on most popular GANs model, including DCGAN, WGAN and WGAN-GP, also wrote a detailed report to discuss their properties. source code
3. Applied basic GAN model to more complicated problem, Image-to-Image Translation, my implementations included cGAN and cycleGAN.

source code
Manually implemented Faster RCNN Deep Network (Tensorflow based) to achieve the object detection and classification task, here we used cifar10 data as the target object and cifar100 data as the background.

Implemented Visual SLAM package and built 3D trajectory as well as 3D point cloud map.

1. Structure From Motion(SFM): implemented Triangulation, PnP and Bundle Adjustment manually with Jacobian for camera state and 3D landmarks estimation and optimization. source code
2. Visual SLAM Package on a Quadrotor: based on SFM structure, succeeded to implement Visual Odometry(VO), applied local BA(Bundle Adjustment) and combined g2O package to increase the SLAM efficiency as well as performance. source code

Implemented some of the most popular used machine learning algorithms and applied them into the real robot cases.

1. Object Detection and Geometry distance estimation based on GMM: trained some GMMs based on the image color data, also combined image processing and geometry algorithms to detetct any read barrel in a robot’s view and estimate its real geometry distance from robot. source code
2. Robot Gesture Recoginition using Hidden Markov Model (HMM): built an algorithm based on the HMM to recognize different robot arm motion gestures. Finally, it was able to classify unknown arm motions within almost real-time. source code
3. Reinforcement Learning using Policy Gradient Methods: implemented several methods for dealing with the Robot Walking on the Frozen Lake problem. The situation can be represented via a Markov Decision Process(MDP) and a strategy for retrieving the frisbee can be obtained using value iteration (VI), policy iteration (PI), and policy gradient optimization (PGO). source code

source code
demo 1 / demo 2 / demo 3 / demo 4

1. Implemented the filter-based 2D SLAM, used IMU data for prediction, Laser-scan data for update, and particle filter for state converge as well as 2D log-odds map optimization.
2. All data were collected on a ground-walkder robot.

source code
demo 1 / demo 2 / demo 3 / demo 4

1. Succeeded in implementing a Kalman filter to track three-dimensional orientation of a hand-handled camera.
2. Given IMU sensor readings from gyroscopes and accelerometers, the algorithm will estimate the underlying 3D orientation by learning the appropriate model parameters from ground truth data given by a Vicon motion capture system.
3. Generated a real-time panoramic image using images captured by the camera and information filtered by the 3D orientation filter.

source code
Java provides several types of numbers, but it does not provide fractions, therefore to implement a Fraction API (Application Programmer’s Interface).

source code
demo 1 / demo 2

1. Utilized human skin color as feature to train GMM model to filter out face region candidates. Combined with edge mask to separate union face regions for better detection.
2. Implemented PCA to construct Eigen Faces dataset and included the third-part package Face++ to improve detection performance. The final accuracy of detection reached 82.6%.
3. Implemented image morphing such as TPS and gradient blending to complete face replacement task.