Heterogeneous SoC-based Improved LWIR Thermal Imaging System for Autonomous Driving
Cheol-Ho Choi
IEEE International Symposium on Circuits and Systems (ISCAS), London, United Kingdom (UK), May 2025
on Writing
@inproceedings{2025_ISCAS,address={London, United Kingdom (UK)},title={Heterogeneous SoC-based Improved LWIR Thermal Imaging System for Autonomous Driving},language={en},booktitle={IEEE International Symposium on Circuits and Systems (ISCAS)},publisher={IEEE},author={Choi, Cheol-Ho},month=may,year={2025},}
An SoC FPGA-based Real-time Image Processor for LWIR Thermal Imaging Systems
Cheol-Ho Choi
IEEE International Symposium on Quality Electronic Design (ISQED’25), San Francisco, United State (US), Apr 2025
on Writing
@inproceedings{2025_ISQED,address={San Francisco, United State (US)},title={An SoC FPGA-based Real-time Image Processor for LWIR Thermal Imaging Systems},language={en},booktitle={IEEE International Symposium on Quality Electronic Design (ISQED'25)},publisher={IEEE},author={Choi, Cheol-Ho},month=apr,year={2025},}
LWIR Thermal Image Processing Algorithm Combining Global and Local Contrast Enhancement for Improving Visibility
Cheol-Ho Choi
IEEE International Conference on Industrial Technology (ICIT), Wuhan, China, Mar 2025
on Writing
@inproceedings{2025_ICIT,address={Wuhan, China},title={LWIR Thermal Image Processing Algorithm Combining Global and Local Contrast Enhancement for Improving Visibility},language={en},booktitle={IEEE International Conference on Industrial Technology (ICIT)},publisher={IEEE},author={Choi, Cheol-Ho},month=mar,year={2025},}
Region-Based Contrast Enhancement for Infrared Thermal Imaging Systems: Parameter-Wise Performance Evaluation and Comparative Analysis
Cheol-Ho Choi
IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, United State (US), Jan 2025
Under Review
@inproceedings{2025_ICCE,address={Las Vegas, United State (US)},title={Region-Based Contrast Enhancement for Infrared Thermal Imaging Systems: Parameter-Wise Performance Evaluation and Comparative Analysis},language={en},booktitle={IEEE International Conference on Consumer Electronics (ICCE)},publisher={IEEE},author={Choi, Cheol-Ho},month=jan,year={2025},}
@article{2024_ACCESS,title={Edge-Based Operation Skip Scheme for Fast Object Detection using Viola-Jones Classifier},language={en},booktitle={IEEE Access},publisher={IEEE},author={Choi, Cheol-Ho and Han, Joonhwan and Oh, Hyun Woo and Cha, Jeongwoo and Shin, Jungho},month=may,year={2024},pages={1-1},}
@inproceedings{2024_ICCE_ASIA_02,address={Da Nang, Vietnam},title={Algorithm for LWIR Thermal Imaging Camera with Minimal Mechanical Shutter Utilization},language={en},booktitle={IEEE International Conference on Consumer Electronics Asia (ICCE-Asia)},publisher={IEEE},author={Kim, Taehyun and Han, Joonhwan and Cha, Jeongwoo and Choi, Hyunmin and Shin, Jungho and Oh, Hyun Woo and Choi, Cheol-Ho and Kim, Eunchong},month=nov,year={2024},}
@inproceedings{2024_ICCE_ASIA_01,address={Da Nang, Vietnam},title={Infrared Thermal Imaging for Embedded Child Presence Detection System: Feasibility and Performance Evaluation},language={en},booktitle={IEEE International Conference on Consumer Electronics Asia (ICCE-Asia)},publisher={IEEE},author={Choi, Cheol-Ho and Hong, Seongtaek and Jeong, Eun Jin and Han, Joonhwan},month=nov,year={2024},}
This paper presents a real-time embedded thermal imaging system architecture for compact, energy-efficient, high-quality imaging utilizing heterogeneous system-on-chip and uncooled infrared focal plane arrays (IRFPAs). In contrast to previous systems that organized separate devices for complex image processing, our system provides integrated image processing support for robust sensor-to-surveillance. We organized the image processing architecture into two algorithm stacks: a non-uniformity correction stack to mitigate the distinctive noise vulnerability of uncooled IRFPAs and an image enhancement stack, which includes contrast enhancement and frame-level temporal noise filters. We optimized the algorithms for domain-specific factors, including asymmetric multiprocessing (AMP), cache organization, single instruction multiple data (SIMD) instructions, and very long instruction word (VLIW) architectures. The implementation on TI TDA3x SoC demonstrates that our system can process 640×480, 60 frames per second (FPS) videos at a peak core load of 57.5% while consuming power less than 2.2 W for the entire system, denoting the possibility of processing the 1280×1024, 30 FPS videos from the state-of-the-art IRFPAs.
@inproceedings{2024_RTCSA,address={Sokcho, Republic of Korea},title={A Compact Real-Time Thermal Imaging System Based on Heterogeneous System-on-Chip},language={en},booktitle={IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA)},publisher={IEEE},author={Oh, Hyun Woo and Choi, Cheol-Ho and Cha, Jeongwoo and Choi, Hyunmin and Shin, Jungho and Han, Joonhwan},month=aug,year={2024},pages={97-107},}
Long-wave infrared-based thermal cameras employing uncooled detectors are extensively utilized in the night vision systems of autonomous driving platforms. In these cameras, contrast enhancement process is necessary to transition from low dynamic range to high dynamic range. Therefore, in this paper, we propose an improved contrast enhancement algorithm utilizing histogram equalization with gamma correction.
@inproceedings{2024_ISOCC,address={Sapporo, Japan},title={Improved Contrast Enhancement Algorithm for Night Vision Systems using Thermal Camera},language={en},booktitle={International Conference on SoC Design Conference (ISOCC)},publisher={IEIE},author={Choi, Cheol-Ho and Cha, Jeongwoo and Han, Joonhwan and Choi, Hyunmin and Shin, Jungho},month=aug,year={2024},pages={56-57},}
In the autonomous driving industry, there is a growing trend to employ long-wave infrared (LWIR)-based uncooled thermal-imaging cameras, capable of robustly collecting data even in extreme environments. Consequently, both industry and academia are actively researching contrast-enhancement techniques to improve the quality of LWIR-based thermal-imaging cameras. However, most research results only showcase experimental outcomes using mass-produced products that already incorporate contrast-enhancement techniques. Put differently, there is a lack of experimental data on contrast enhancement post-non-uniformity (NUC) and temperature compensation (TC) processes, which generate the images seen in the final products. To bridge this gap, we propose a histogram equalization (HE)-based contrast enhancement method that incorporates a region-based clipping technique. Furthermore, we present experimental results on the images obtained after applying NUC and TC processes. We simultaneously conducted visual and qualitative performance evaluations on images acquired after NUC and TC processes. In the visual evaluation, it was confirmed that the proposed method improves image clarity and contrast ratio compared to conventional HE-based methods, even in challenging driving scenarios such as tunnels. In the qualitative evaluation, the proposed method demonstrated upper-middle-class rankings in both image quality and processing speed metrics. Therefore, our proposed method proves to be effective for the essential contrast enhancement process in LWIR-based uncooled thermal-imaging cameras intended for autonomous driving platforms.
@article{2024_SENSORS,title={Contrast Enhancement Method using Region-based Dynamic Clipping Technique for LWIR-based Thermal Camera of Night Vision Systems},language={en},journal={Sensors},publisher={MDPI},author={Choi, Cheol-Ho and Han, Joonhwan and Cha, Jeongwoo and Choi, Hyunmin and Shin, Jungho and Kim, Taehyun and Oh, Hyun Woo},month=jun,year={2024},volume={24},number={12},}
Machine learning approaches are preferred over deep learning in embedded systems due to their resource efficiency. The widely adopted Viola-Jones method and related algorithms are selected for their high detection accuracy and reasonable processing speed. However, a limitation arises as processing time increases with additional classification iterations based on sub-window operations. To address this issue, we propose an enhanced object detection algorithm that incorporates the Viola-Jones method with edge component calibration and an edge-based operation skip scheme. The introduction of edge component calibration ensures detection performance comparable to conventional methods. This scheme, relying on edge values, significantly reduces unnecessary computations in the background, leading to a marked decrease in classification operations compared to conventional methods. Visual comparisons in experimental results demonstrate that our method increases the detection precision factor while maintaining recall. In terms of classification operations, our approach reduces their number by 31.38% to 85.78% compared to conventional methods. In simpler terms, our method improves processing speed by minimizing classification operations, making it well-suited for embedded systems with limited resource utilization.
@inproceedings{2024_AICAS,address={Abu Dhabi, United Arab Emirates (UAE)},title={Fast Object Detection Algorithm using Edge-based Operation Skip Scheme with Viola-Jones Method},language={en},booktitle={IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)},publisher={IEEE},author={Choi, Cheol-Ho and Han, Joonhwan and Cha, Jeongwoo and Shin, Jungho and Oh, Hyun Woo},month=apr,year={2024},pages={199--203},}
Embedded stereo vision systems based on traditional approaches often require a disparity refinement process to enhance image quality. Weighted median filter (WMF)-based processors are commonly employed for their excellent refinement performance. However, when implemented on a field-programmable gate array (FPGA), WMF-based processors face a trade-off between hardware resource utilization and refinement performance. To address this trade-off, we previously proposed a new disparity refinement processor based on the hybrid max-median filter (HMMF). However, our earlier work did not guarantee flawless operation in large occluded and texture-less regions, particularly in areas with numerous holes. In order to overcome this limitation of conventional processors, we proposed a cell-based disparity refinement processor. This processor extends our previous HMMF-based disparity refinement processor. To evaluate its refinement performance, we conducted experiments using four types of publicly available stereo datasets. When comparing refinement performance, our proposed processor outperforms conventional processors when using the KITTI 2012 and 2015 stereo benchmark datasets. Additionally, the results demonstrate that our proposed processor exhibits superior refinement performance when applied to the Cityscapes and StereoDriving datasets in comparison to conventional processors. Furthermore, when considering hardware resource utilization, our proposed processor demonstrates lower resource requirements than conventional processors when implemented on an FPGA. Therefore, our proposed disparity refinement processor is well-suited for the disparity refinement process in stereo vision systems that require cost-effectiveness and high performance.
@article{2023_ACCESS,title={Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision Systems},volume={11},issn={2169-3536},doi={10.1109/ACCESS.2023.3338649},language={en},journal={IEEE Access},author={Choi, Cheol-Ho and Oh, Hyun Woo and Han, Joonhwan and Shin, Jungho},month=dec,year={2023},pages={138122--138140},}
This paper presents an integrated image processor architecture designed for real-time interfacing and processing of high-resolution thermal video obtained from an uncooled infrared focal plane array (IRFPA) utilizing a modern system-onchip field-programmable gate array (SoC FPGA). Our processor provides a one-chip solution for incorporating non-uniformity correction (NUC) algorithms and contrast enhancement methods (CEM) to be performed seamlessly. We have employed NUC algorithms that utilize multiple coefficients to ensure robust image quality, free from ghosting effects and blurring. These algorithms include polynomial modeling-based thermal drift compensation (TDC), two-point correction (TPC), and run-time discrete flat field correction (FFC). To address the memory bottlenecks originating from the parallel execution of NUC algorithms in real-time, we designed accelerators and parallel caching modules for pixel-wise algorithms based on a multi-parameter polynomial expression. Furthermore, we designed a specialized accelerator architecture to minimize the interrupted time for run-time FFC. The implementation on the XC7Z020CLG400 SoC FPGA with the QuantumRed VR thermal module demonstrates that our image processing module achieves a throughput of 60 frames per second (FPS) when processing 14-bit 640×480 resolution infrared video acquired from an uncooled IRFPA.
@inproceedings{2023_DSD_02,address={Golem, Albania},title={An SoC FPGA-based Integrated Real-time Image Processor for Uncooled Infrared Focal Plane Array},language={en},booktitle={Euromicro Conference on Digital System Design (DSD)},publisher={IEEE},author={Oh, Hyun Woo and Choi, Cheol-Ho and Cha, Jeongwoo and Choi, Hyunmin and Han, Joonhwan and Shin, Jungho},month=sep,year={2023},pages={660--668},}
In embedded stereo vision systems based on semiglobal matching, the matching accuracy of the initial disparity map can be degraded because of various factors. To solve this problem, weighted median-based disparity refinement hardware architectures are utilized to improve the matching accuracy. However, for the conventional hardware architectures, there is a trade-off between hardware resource utilization and refinement performance when they are implemented on a field programmable gate array (FPGA). Therefore, in this paper, we propose a hybrid max-median filter and its hardware architecture to improve the refinement performance and reduce hardware resource utilization. To evaluate the refinement performance, we used two public stereo datasets. When using the various window sizes for KITTI 2012 and 2015 stereo benchmark datasets, the proposed hardware architecture showed better matching accuracy performance compared with the conventional hardware architectures. In terms of the hardware resource utilization, when implemented on an FPGA, the proposed hardware architecture has low requirements for all types of hardware resources. That is, the proposed hardware architecture overcomes the trade-off between hardware resource utilization and refinement performance.
@inproceedings{2023_DSD_01,address={Golem, Albania},title={Disparity Refinement Processor Architecture utilizing Horizontal and Vertical Characteristics for Stereo Vision Systems},language={en},booktitle={Euromicro Conference on Digital System Design (DSD)},publisher={IEEE},author={Choi, Cheol-Ho and Oh, Hyun Woo},month=sep,year={2023},pages={220--226},}
The growing significance of the security and human management fields attracts active research related to face detection and recognition systems. Among these face detection techniques based on machine learning, Haar cascade classifiers are widely used because of their high accuracy for human frontal faces. However, the Haar cascade classifiers have a limitation in that the processing time increases as the number of false positives increases because they detect human faces based on the sub-window operation. Therefore, in this paper, a preprocessing method based on a 2D Haar discrete wavelet transform is proposed for face detection. The proposed method improves the processing speed by reducing the number of false positives through a vertical component calibration process using the vertical and horizontal components. The results of the face detection experiments that use a public test dataset comprising 2,845 images showed that the proposed method improved the processing speed by 32.05% and reduced the number of false positives by 25.46%, compared with those of the histogram equalization that shows the best performance case among conventional filter-based pre-processing methods. In addition, the performance of the proposed method is similar to those of conventional image contraction-based methods. In an experiment using a private dataset, the proposed method showed a 53.85% reduction in the total number of false positives compared with that of the Gaussian filter while maintaining the total number of true positives. The F1 score of the proposed method shows a 1.39% improvement compared with those of Lanczos-3 that shows the best performance case.
@article{2022_HCIS,title={Face Detection using Haar Cascade Classifiers based on Vertical Component Calibration},issn={2192-1962},author={Choi, Cheol-Ho and Kim, Junghwan and Hyun, Jongkil and Kim, Younghyeon and Moon, Byungin},journal={Human-centric Computing and Information Sciences (HCIS)},volume={12},number={11},month=mar,year={2022},pages={1--17},}
Drivable area detection, one of the main functions of advanced driver assistance systems, means detecting an area where a vehicle can safely drive. The drivable area detection is closely related to the safety of the driver and it requires high accuracy with real-time operation. To satisfy these conditions, V-disparity-based method is widely used to detect a drivable area by calculating the road disparity value in each row of an image. However, the V-disparity-based method can falsely detect a non-road area as a road when the disparity value is not accurate or the disparity value of the object is equal to the disparity value of the road. In a road environment including vegetation, such as a highway and a country road, the vegetation area may be falsely detected as the drivable area because the disparity characteristics of the vegetation are similar to those of the road. Therefore, this paper proposes a drivable area detection method and hardware architecture with a high accuracy in road environments including vegetation areas by reducing the number of false detections caused by V-disparity characteristic. When 289 images provided by KITTI road dataset are used to evaluate the road detection performance of the proposed method, it shows an accuracy of 90.12% and a recall of 97.96%. In addition, when the proposed hardware architecture is implemented on the FPGA platform, it uses 8925 slice registers and 7066 slice LUTs.
@article{2022_KTSDE,title={Filtering-based Method and Hardware Architecture for Drivable Area Detection in Road Environment Including Vegetation},author={Kim, Younghyeon and Ha, Jiseok and Choi, Cheol-Ho and Moon, Byungin},journal={KIPS Transactions on Software and Data Engineering (KTSDE)},volume={11},number={1},pages={51--58},month=jan,year={2022},publisher={KIPS},}
In stereo vision systems, mismatching can occur frequently in specific regions containing noise and high-frequency components, such as checkered patterns. Therefore, this paper proposes a preprocessing method and architecture based on a 2-D Haar filter to improve the matching accuracy of disparity map by reducing the high-frequency and noise components. In this paper, the disparity map is computed by the semi-global matching (SGM) method, and separable weighted median filter is adopted as postprocessing. The SGM with proposed method reduces the average number of mismatching pixels by 3.5233% in non-occlusion condition and 3.2647% in occlusion condition compared to that with a 2-D Gaussian filter when using the KITTI 2015 stereo dataset. The proposed method is suitable for embedded stereo vision systems that requires high matching accuracy with reasonable resource overhead.
@inproceedings{2021_ISOCC,address={Jeju, Republic of Korea},title={Haar Filter Hardware Architecture for the Accuracy Improvement of Stereo Vision Systems},author={Choi, Cheol-Ho and Kim, Younghyeon and Ha, Jiseok and Moon, Byungin},booktitle={International SoC Design Conference (ISOCC)},pages={401--402},month=oct,year={2021},organization={IEEE},}
Face recognition applications are being widely studied owing to their extensive usability in the field of computer vision. However, processing an entire image requires a considerable amount of time. To reduce the processing time, several algorithms that extract only the face from the image during pre-processing are studied. Haar classifiers are extensively used for the hardware implementation of face detection algorithms that improve the processing speed of face classification. This paper proposes a Haar classifier based face detection architecture that removes unnecessary iterations during classification to further improve the processing speed. The proposed architecture improves the processing speed by 4.46% compared to that of conventional Haar classifier based face detection architectures, for face detection using a VGA image with 30 faces. The proposed architecture tends to improve the processing speed as the number of faces in the image increases while matching the detection accuracy of conventional methods. Additionally, this architecture can be widely applied to classification algorithms that are based on iterations.
@inproceedings{2021_ISCAS,address={Daegu, Republic of Korea},title={Hardware Architecture of a Haar Classifier based Face Detection System using a Skip Scheme},author={Hyun, Jongkil and Kim, Junghwan and Choi, Cheol-Ho and Moon, Byungin},booktitle={IEEE International Symposium on Circuits and Systems (ISCAS)},pages={1--4},month=may,year={2021},organization={IEEE},}
The heartbeat detection using the scaling function of Wavelet transform is proposed for a Doppler radar sensor. The conventional methods such as the fast-Fourier transform and the autocorrelation show the respiration rate and the heartbeat from the raw data of the radar sensors acquiring for a sufficient sampling time. The methods have the limit to detect the biometric information that varies with real-time because they only show the overall statistical information of the sampled data. In the proposed method, the scaling function in the Daubechies wavelet transform can be used to accurately find out the periodicity of radar signals for detecting heartbeat varying in real-time. The results of the signal processing using the radar signals acquired for 3 min results show that the proposed method lowered a mean error rate of 2.5% and a SD of 2.0% compared with the method using the wavelet function. The proposed method in the measurement for 1 minute using the radar sensor also showed the lowest mean error rate of 3.8% and the low SD of 3.2% using the contact sensor as the reference among various signal processing methods including auto-correlation and peak detection with filtering.
@article{2019_MOTL,title={Heartbeat Detection using a Doppler Radar Sensor based on the Scaling Function of Wavelet Transform},author={Choi, Cheol-Ho and Park, Jae-Hyun and Lee, Ha-Neul and Yang, Jong-Ryul},journal={Microwave and Optical Technology Letters (MOTL)},volume={61},number={7},pages={1792--1796},month=jul,year={2019},publisher={Wiley Online Library},}