×4
Bicubic EDSR [16] RBPN [7] Ours HR
Figure 3.1: Qualitative results. Zoom in to see better visualization.
3.5 Ablation study
We adopt the unidirectional ConvLSTM as the simplest baseline. As shown in the Tab. 3.2, the temporal information is important since the model performance is worse when the memory cells in ConvLSTM are disabled. As the cardiac MRI video is cyclic, we can refresh the memory by feeding n successive frames. Accordingly, we analyze the relation between n and model performance. The result in Fig 3.2b turns out that the network significantly improves as the updated frame number increases. Moreover, the forward and backward information is shown to be useful and complementary for recovering the lost details. In Sec. 2.2, we exploit the knowledge of the cardiac phase to better fuse the bidirectional information. The result in Tab. 3.2 reveals that the phase fusion module can leverage the bidirectional temporal features more effectively. Besides, we explore the influence of the total number of refinement stages Ω in the residual of residual learning.
It can be observed from Fig. 3.2c that the reconstruction performance is improved as the total refinement stages continue to increase. The possible reason for the saturation or degradation of the overall performance when Ω equals to 3 or 4 is overfitting.
Table 3.2: Ablation study. Memory: the memory cells in the ConvLSTM [28] are acti-vated; Updated memory: the memory cells are updated by feeding n consecutive frames;
Bidirection: bidirectional ConvLSTM is adopted; Phase fusion module and Residual of residual learning: the proposed components are adopted.
Memory Updated memory
(n = 6) Bidirection Phase fusion module Residual of residual learning
(Ω = 2) CardiacPSNR/CardiacSSIM
30.3 30.4 30.5 30.6 30.7 30.8 30.9 CardiacPSNR
(a) Efficiency vs performance on DSB15SR dataset for scale ×4. (FPS: processed frames per second)
0 1 2 3 4 5 6 Number of updated frames (n) 30.07 Number of updated frames (n) 0.8561
(b) Analysis of the update frame number n.
0 1 2 3 4
Total refinement stages ( ) 30.27
Total refinement stages ( ) 0.8632
(c) Analysis of total refinement stages Ω.
Figure 3.2: Experimental analysis. (a) Our network outperforms other baselines with fewer parameters and higher FPS. (b) The performance is progressively enhanced as n increases, which indicates that the prior sequence can provide useful information. (c) The performance can be improved with Ω increasing.
Chapter 4 Conclusion
In this work, we define the cyclic cardiac MRI video super-resolution problem which has not yet been completely solved to our best knowledge. To tackle this issue, we bring the cardiac knowledge into our network and employ the residual of residual learning to train in the progressive refinement manner, which enables the model to generate sharper results with fewer model parameters. In addition, we build large-scale datasets and introduce car-diac metrics for this problem. Through extensive experiments, we demonstrate that our network outperforms the state-of-the-art baselines qualitatively and quantitatively. Most notably, we carry out the external evaluation, which indicates our model exhibits good generalization behavior. We believe our approach can be seamlessly applied to other modalities such as computed tomography angiography and echocardiography.
Bibliography
[1] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P.-A. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester, et al. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE transactions on medical imaging, 37(11):2514–2525, 2018.
[2] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4733–4742, 2016.
[3] Y. Chen, F. Shi, A. G. Christodoulou, Y. Xie, Z. Zhou, and D. Li. Efficient and accurate mri super-resolution using a generative adversarial network and 3d multi-level densely connected network. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 91–99. Springer, 2018.
[4] Y. Chen, Y. Xie, Z. Zhou, F. Shi, A. G. Christodoulou, and D. Li. Brain mri super resolution using 3d deep densely connected neural networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 739–742. IEEE, 2018.
[5] C. Dong, C. C. Loy, K. He, and X. Tang. Image super-resolution using deep convo-lutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
[6] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pages 64–72, 2016.
[7] M. Haris, G. Shakhnarovich, and N. Ukita. Recurrent back-projection network for video super-resolution. arXiv preprint arXiv:1903.10128, 2019.
[8] J.-B. Huang, A. Singh, and N. Ahuja. Single image super-resolution from trans-formed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2015.
[9] Y. Huang, W. Wang, and L. Wang. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In Advances in Neural Information Processing Systems, pages 235–243, 2015.
[10] Y. Jo, S. Wug Oh, J. Kang, and S. Joo Kim. Deep video super-resolution network us-ing dynamic upsamplus-ing filters without explicit motion compensation. In Proceed-ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3224–3232, 2018.
[11] A. Jog, A. Carass, and J. L. Prince. Self super-resolution for magnetic resonance images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 553–560. Springer, 2016.
[12] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[13] A. Lalande, N. Salve, A. Comte, M.-C. Jaulent, L. Legrand, P. Walker, Y. Cottin, J.-E. Wolf, and F. Brunotte. Left ventricular ejection fraction calculation from auto-matically selected and processed diastolic and systolic frames in short-axis cine-mri.
Journal of Cardiovascular Magnetic Resonance, 6(4):817–827, 2004.
[14] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution us-ing a generative adversarial network. In Proceedus-ings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
[15] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In Artificial intelligence and statistics, pages 562–570, 2015.
[16] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 136–144, 2017.
[17] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[18] M. Montemerlo, S. Thrun, D. Koller, B. Wegbreit, et al. Fastslam: A factored so-lution to the simultaneous localization and mapping problem. Aaai/iaai, 593598, 2002.
[19] C.-H. Pham, A. Ducournau, R. Fablet, and F. Rousseau. Brain mri super-resolution using deep 3d convolutional networks. In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pages 197–200. IEEE, 2017.
[20] M. S. Sajjadi, R. Vemulapalli, and M. Brown. Frame-recurrent video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6626–6634, 2018.
[21] M. Salerno, B. Sharif, H. Arheden, A. Kumar, L. Axel, D. Li, and S. Neubauer.
Recent advances in cardiovascular magnetic resonance: techniques and applications.
Circulation: Cardiovascular Imaging, 10(6):e003951, 2017.
[22] J. Shi, Q. Liu, C. Wang, Q. Zhang, S. Ying, and H. Xu. Super-resolution recon-struction of mr image with a novel residual learning network algorithm. Physics in Medicine & Biology, 63(8):085011, 2018.
[23] L. Tautz, O. Friman, A. Hennemuth, A. Seeger, and H.-O. Peitgen. Automatic de-tection of a heart roi in perfusion mri images. In Bildverarbeitung für die Medizin 2011, pages 259–263. Springer, 2011.
[24] L. The National Heart and B. Institute. Data science bowl cardiac challenge data, 2015.
[25] F. von Knobelsdorff-Brenkenhoff, G. Pilz, and J. Schulz-Menger. Representation of cardiovascular magnetic resonance in the aha/acc guidelines. Journal of Cardiovas-cular Magnetic Resonance, 19(1):70, 2017.
[26] F. von Knobelsdorff-Brenkenhoff and J. Schulz-Menger. Role of cardiovascular magnetic resonance in the guidelines of the european society of cardiology. Journal of Cardiovascular Magnetic Resonance, 18(1):6, 2015.
[27] X. Wang, K. C. Chan, K. Yu, C. Dong, and C. Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
[28] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo. Convo-lutional lstm network: A machine learning approach for precipitation nowcasting.
In Advances in neural information processing systems, pages 802–810, 2015.
[29] H.-y. Xu, Z.-g. Yang, Y.-k. Guo, K. Shi, X. Liu, Q. Zhang, L. Jiang, and L.-j. Xie.
Volume-time curve of cardiac magnetic resonance assessed left ventricular dysfunc-tion in coronary artery disease patients with type 2 diabetes mellitus. BMC cardio-vascular disorders, 17(1):145, 2017.
[30] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127(8):1106–1125, 2019.
[31] C. Zhao, A. Carass, B. E. Dewey, and J. L. Prince. Self super-resolution for magnetic resonance images using deep networks. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pages 365–368. IEEE, 2018.
[32] H. Zhao, O. Gallo, I. Frosio, and J. Kautz. Loss functions for neural networks for image processing. arXiv preprint arXiv:1511.08861, 2015.