Dr Lingbo Cheng

Intraoperative situational awareness is critical for clinicians when performing complex surgeries and therapies. Augmented-reality (AR) and haptic virtual fixtures (VF) technologies can be used to increase situational awareness while still keeping the surgeon-in-the-loop. These two technologies can be used independently or together to provide various levels of assistance. 3D AR technologies provide enhanced visual feedback for the clinician, for instance, giving them the ability to "see" surgical instruments inside of tissue. Robotic assistant devices with haptic VF provide physical resistance or assistance in real-time for the clinician. Haptic VF systems can be used to enforce "no-fly zones" with varying degrees of resistance to allow the clinician to avoid damaging sensitive tissue. The assistance provided by haptic VF systems can reduce the strain on a clinician when performing a lengthy surgery, can move the surgical tool to compensate for changes in patient pose, and can increase the precision with which a clinician can perform various surgical tasks.

This chapter will explore certain applications of control engineering for improving the performance of robotic and mechatronic systems for therapies and surgeries, we will consider prostate brachytherapy and beating-heart surgery as two examples. In permanent implant brachytherapy, needles loaded with radioactive seeds are used to reach planned locations in the prostate, where the seeds are deployed. Accurate seed placement is a key factor that influences the effectiveness of the procedure. However, current manual techniques can place seeds with an accuracy of only about 5. mm. This is a substantial error given the average prostate size and narrows the scope of brachytherapy to primarily treating the entire prostate gland for patients with localized prostate cancer. In the context of mechatronic systems for precisely steering a needle toward its intended location in a controlled manner, we will review the related observer and controller design methods.While operating on a beating heart would offer many benefits to patients, performing a surgical task on the beating heart requires superhuman skills as the surgeon must manually track the heart's motion while performing the surgical task. With advances in surgical robotics, we can now envision a robot-assisted surgical system that first synchronizes the surgical robots motion with the beating heart's motion and then lets the surgeon operate through teleoperation on a seemingly motionless point on the heart. For such a system that relies on image guidance, the heart motion prediction and predictive feedback control to enable beating-heart surgery, we will again discuss how observers and controllers can be designed to improve the robotic system performance.

Compared to conventional arrested heart surgery, beating-heart surgery is promising as the advantages of eliminating adverse effects caused by a heart-lung bypass machine and enabling intraoperative evaluation of heart motion. However, the fast motion of the heart introduces a significant challenge for beating-heart surgery. In this paper, a teleoperation system, which employs an impedance control for the master robot and an ultrasound image-based position control for the slave robot (surgical robot), is proposed to achieve non-oscillatory force feedback and heart motion compensation, respectively. Specifically, an impedance model is designed for the master robot to provide the human operator (surgeon) with non-oscillatory haptic feedback. To compensate for the beating heart's motion, ultrasound imaging is used to obtain the position of the point of interest (POI) on the heart tissue. As the use of ultrasound imaging introduces non-negligible time delay caused by image acquisition and processing, a recurrent neural network (NN)-based physiological organ motion predictor is proposed. The predicted POI position is used to control the slave robot to automatically compensate for the beating heart's motion. The proposed method is validated through experiments. The proposed control strategy with NN-based heart motion predictor is compared to the other two strategies without heart motion predictor and with an extended Kalman filter (EKF)-based heart motion predictor. The experimental results present that the proposed strategy with NN algorithm shows significant advantages (higher synchronization accuracy and relatively steady slave-heart contact force) over the other two strategies.

In December 2019, an outbreak of novel coronavirus pneumonia occurred, and subsequently attracted worldwide attention when it bloomed into the COVID-19 pandemic. To limit the spread and transmission of the novel coronavirus, governments, regulatory bodies, and health authorities across the globe strongly enforced shut down of educational institutions including medical and dental schools. The adverse effects of COVID-19 on dental education have been tremendous, including difficulties in the delivery of practical courses such as restorative dentistry. As a solution to help dental schools adapt to the pandemic, we have developed a compact and portable teaching-learning platform called DenTeach. This platform is intended for remote teaching and learning pertaining to dental schools at these unprecedented times. This device can facilitate fully remote and physical-distancing-aware teaching and learning in dentistry. DenTeach platform consists of an instructor workstation (DT-Performer), a student workstation (DT-Student), advanced wireless networking technology, and cloud-based data storage and retrieval. The platform procedurally synchronizes the instructor and the student with real-time video, audio, feel, and posture (VAFP). To provide quantitative feedback to instructors and students, the DT-Student workstation quantifies key performance indices (KPIs) related to a given task to assess and improve various aspects of the dental skills of the students. DenTeach has been developed for use in teaching, shadowing, and practice modes. In the teaching mode, the device provides each student with tactile feedback by processing the data measured and/or obtained from the instructor's workstation, which helps the student enhance their dental skills while inherently learning from the instructor. In the shadowing mode, the student can download the augmented videos and start watching, feeling, and repeating the tasks before entering the practice mode. In the practice mode, students use the system to perform dental tasks and have their dental performance skills automatically evaluated in terms of KPIs such that both the student and the instructor are able to monitor student¿s work. Most importantly, as DenTeach is packaged in a small portable suitcase, it can be used anywhere by connecting to the cloud-based data storage network to retrieve procedures and performance metrics. This paper also discusses the feasibility of the DenTeach device in the form of a case study. It is demonstrated that a combination of the KPIs, video views, and graphical reports in both teaching and shadowing modes effectively help the student understand which aspects of their work needs further improvement. Moreover, the results of the practice mode over 10 trials have shown significant improvement in terms of tool handling, smoothness of motion, and steadiness of the operation.

In this paper, a semi-autonomous robot control system is developed for 3D robotic tracking of the complex physiological organ motion introduced by respiration and heartbeat in cardiac surgery. The same control system enables the surgeon's hand to perceive the non-oscillatory portion of the surgical robot-heart tissue interaction force. The semi-autonomous surgical system includes a slave surgical robot which can compensate for the physiological organ motion automatically and a master robot (user interface) which is manipulated by the surgeon to provide task commands to the surgical robot. The proposed impedance control method for the surgical robot only needs the frequency range of the physiological motion to synchronize the surgical instrument with the organ motion automatically. Another reference impedance model for the master robot is designed to provide non-oscillatory force feedback to the surgeon. A usability study emulating the motion requirements of tissue ablation is carried out. Experimental results are presented to show the effectiveness of the proposed method by comparing the results to the manual compensation method.

A neural-network-based heart motion prediction method is proposed for ultrasound-guided beating-heart surgery to compensate for time delays caused by ultrasound (US) image acquisition and processing. Such image processing is needed for tracking heart tissue in US images, which is itself a requirement for beating-heart surgery. Once the heart tissue is tracked in US images, a recurrent neural network (NN) is employed to learn how to predict the motion of the tracked heart motion in order to compensate for the delays introduced in the initial US image processing step. To verify the feasibility of predicting both simple and complex heart motions, the NN is tested with two types of heart motion data: (i) fixed heart rate and maximum amplitude, and (ii) varying heart rate and maximum amplitude. Also, the NN was tested for different prediction horizons and showed effectiveness for both small and large delays. The heart motion prediction results using NN are compared to the results using an extended Kalman filter (EKF) algorithm. Using NN, the mean absolute error and the root mean squared error between the predicted and the actually tracked heart motions are roughly 60% smaller than those achieved by using the EKF. Moreover, the NN is able to predict the heart position up to 1000 ms in advance, which significantly exceeds the typical US image acquisition/processing delays for this application (160 ms in our tests). Overall, the NN predictor shows significant advantages (higher accuracy and longer prediction horizon) compared to the EKF predictor.

Flexible and lightweight surgical tools have the potential to significantly increase the dexterity of mechatronics-assisted surgical systems for minimally invasive surgeries. However, the control of a mechatronics-assisted system with the link and joint flexibility is quite challenging and needs to be studied. In this paper, a bilateral impedance-controlled master-slave teleoperation system is considered, where the slave (surgical) robot is flexible. Two reference impedance models are designed for the master and slave robots to control the mechatronics-assisted system. Also, depending on different feedback and feedforward signals, four cases are distinguished. To obtain better transparency of the system, the tuning rules for the impedance parameters for each case are presented and the corresponding transparency measures are analyzed and compared. As a result, by appropriately adjusting the impedance model parameters, ideal position and force tracking can be attained for a teleoperation system with a flexible surgical robot. The theoretical findings are validated in simulations.