Introduction
Recently, I worked on an OpenCV face recognition project. It utilizes OpenCV and the contrib module APIs, with the Haar Cascade classifier for face detection and the LBP algorithm for extracting facial features. The extracted data is stored in an SQLite3 database, and clock-in records can also be exported as CSV format tables. The face recognition process involves the following steps:
WARNING
📌 1. Install QT5.12.12, OpenCV, and contrib modules on Ubuntu, and import the code, which can also be used on the Ubuntu side. 2. Compile the Linux system, configure the MIPI screen, and flash the system onto the Taishan RK3566. 3. Cross-compile QT5.12.12, OpenCV, and contrib modules to generate the underlying runtime libraries for RK3566. 4. Use the cross-compiled libraries to configure the QT application, cross-compile the face recognition application as an ARM application, and copy the libraries and application to the Taishan RK3566 using USB or NFS. 5. Launch the application and configure the environment.
This face recognition system simply calls OpenCV's built-in functions. Due to my limited capability, I didn't use GPU or NPU. When using the CPU, the performance was sluggish, and I hope someone more experienced will share a method for implementing deep learning face recognition with GPU and NPU, such as using MTCNN for MobileFaceNet.
Some might wonder why the video processes images rather than videos. In fact, face recognition, in essence, processes images. The video is simply a series of images, and I’m extracting a frame from the video for processing. However, I haven't implemented live detection, which can be done using infrared cameras or depth cameras without needing additional algorithms.
Development Environment: Ubuntu 22.04, GCC 9, QT5.12.12, OpenCV 4.6, Linaro GCC 7.5 cross-compiler Hardware: LCSC Taishan-RK3566-Linux Dev Board (2GB RAM + 16GB storage), UVC camera, 1280×800 screen, USB drive The files used in this project can be downloaded from Gitee: https://gitee.com/LJ96324/rk3566.git
See the final video for application results
Note: The face-test application is provided with an optimized version, but I only offer the unoptimized code for others to improve. The final optimized version of the code is not convenient to share.
1. Install QT5.12.12 on Ubuntu, compile OpenCV and contrib modules, and import code to work on Ubuntu.
My code was written on Ubuntu, and to compile or modify it, you need to install QT. For installation, you can refer to this tutorial: https://blog.csdn.net/dengjin20104042056/article/details/131705766
If you don't need to modify anything, you can skip the installation of QT and directly use the executable binary files I provide.
I won’t go into the details of QT installation here, it's pretty straightforward. I’ll briefly mention OpenCV installation steps.
WARNING
📌 First, install dependencies. You can refer to the following links or use my brief tutorial.
For installing OpenCV and contrib modules, refer to the installation sources (the links are in the source download):
https://blog.csdn.net/qq_19449259/article/details/127701812
https://blog.csdn.net/qq_26943851/article/details/104434130
Brief Tutorial:
sudo apt install cmake-qt-guisudo apt install mingw-w64sudo apt install cmakesudo apt install build-essential libgtk2.0-dev libavcodec-dev libavformat-dev libjpeg.dev libtiff4.dev libswscale-dev libjasper-devsudo apt install git
After installation, mkdir bulid to create the build directory. Open a terminal in the build directory and type cmake-gui to enter the cmake configuration interface. Select the default Makefile configuration.
Search for opencv_extra_modules_path and select the contrib module's module directory.
The default installation directory is in /usr, and you can change it by searching for install. If you don't modify it, you can directly click configure, then generate to create the Makefile for compiling.
Type sudo make -j12 to start compiling OpenCV and the contrib module. After compilation, use sudo make install to install it.
Note that by default, it is installed under /usr. This is for virtual machine use, and installing here is fine because the default directory is easy to locate. If anyone wants to install it in a different directory, you can change it. However, the OpenCV header files and library paths in the QT pro file must also be modified accordingly.
2. Compile the Linux system, configure the MIPI screen, and burn the system to the LCSC Taishan-RK3566-Linux Dev Board
Simply follow the tutorial from the "Pangniu Phone" guide to modify the device tree for the screen and compile and burn the system as described
I am using the Buildroot system.
3. Cross-compile QT, OpenCV, and the contrib module to get the underlying runtime libraries for the 3566 (can be downloaded directly, skipping this step)
For cross-compiling QT, there is a Taishan board tutorial available: Qt5 Porting
For OpenCV cross-compilation:
Delete the contents of the OpenCV build directory and use the cmake-gui command again. Since this is for cross-compiling, select a custom compiler.
Delete the contents of the OpenCV build directory and use the cmake-gui command again. Since this is for cross-compiling, select a custom compiler.
Choose the contrib module's module directory.
Here, you must change the installation directory. This is necessary because if you don’t, it will overwrite the previously installed x86 libraries, making them unusable on Ubuntu.
Click configure, then generate to create the Makefile for compiling.
Start the compilation with make -j12, and after compilation, use make install to install it. No need for sudo if the root directory files are not involved.
Once compiled, you can check the file types to confirm they are for ARM aarch64.
WARNING
📌 While cross-compiling OpenCV, you might encounter errors related to #define _FILE_OFFSET_BITS 64 definition and the undefined reference to 'png_do_expand_palette_rgb8_neon' error. You can resolve this by following the solutions provided in the following blog posts:
5. Configure QT Application with Cross-compiled Libraries, Cross-compile Face Recognition Application for ARM, and Transfer Libraries and Application to the LCSC Taishan-RK3566-Linux Dev Board
Download and Unzip OpenCV Cross-compiled Files: Download the OpenCV cross-compilation files and extract them to the /opt/arm-opencv directory.
Configure QT: If you don't want to cross-compile the QT source code, you can use the precompiled QT package I provided. Unzip it into the /opt/arm-qt5.12.12 directory. If the directory doesn't exist, create it. This is because the cross-compilation files for QT come with path encoding, and placing them in other directories may not work. I chose /opt to avoid username-related issues on different machines.
Open QT Creator and Configure the Cross-compilation Toolchain: Open QT Creator and configure the cross-compilation toolchain for your QT applications. This will allow you to use this toolchain for other QT development in the future.
Configure qmake and Compiler:
After configuring qmake and the compiler, you can add the toolchain.
Once the toolchain is added, you can import your code and compile the application.
Copy the QT and OpenCV directories to /opt on the LCSC Taishan-RK3566-Linux Dev Board.
Copy the font package to /usr/share on the Taishan board and unzip it.
6. Start the Application and Configure the Environment
Setting up the environment is the most challenging part of this tutorial, requiring numerous trial-and-error attempts, but it can be summarized fairly simply.
WARNING
📌 Issue 1: Cannot open the camera. You need to delete all pre-existing video nodes on the LCSC Taishan device. Once the UVC is connected, video9 and video10 will appear. Link these to video0 and video1 as shown below:
rm -rf /dev/video\*
ln -s /dev/video9 /dev/video0
ln -s /dev/video10 /dev/video12
3
4
5
WARNING
📌 Issue 2: Font and OpenCV library issues.
Edit the vi /etc/profile, adding the following lines at the end. Then either source the profile or restart the system.
export QT_DIR=/opt/arm-qt5.12.12
export PATH=$QT_DIR/bin:$PATH
export LD_LIBRARY_PATH=$QT_DIR/lib:$LD_LIBRARY_PATH
export QT_QPA_PLATFORM=linuxfb
export QT_PLUGIN_PATH=$QT_DIR/plugins
export QT_QPA_FONTDIR=/usr/share/fonts/truetype/arphic/2
3
4
5
6
After this, you’re all set! Run the application by entering ./face-test.
7. Demo Video
8. Quickstart Version (For Immediate Use)
- Flash the buildroot or Ubuntu (if there’s screen flickering, kill the desktop process). I haven't solved the Debian issue yet.
Extract arm-qt5.tar.gz and arm-opencv.tar.gz to the /opt/ directory.
Extract fonts.tar.gz to the usr/share/ directory.
Use the following command to extract each archive:gzip -d xxx.tar.gz tar -xavf xxx.tar
Then, follow the 6. Start the Application and Configure the Environment steps, and execute as instructed.