ESP32S3 Running OpenMV for Image Recognition | LC Development Board Tech Document Center

During the development and implementation of this project, we referenced and built upon the source code publicly released by the GitHub developer Kevincoooool (Kushi DIY). The project was completed through secondary editing and processing by internal tester user Huang Caiying. Here we specifically note that the original copyright of the source code belongs to Kevincoooool, and the original copyright of the article belongs to Huang Caiying. We express our deep respect and gratitude to the developer Kevincoooool and the user Huang Caiying for their open-sharing spirit and contributions to the open-source community.

Click to jump

Open-source culture provides fertile soil for technological innovation and knowledge sharing, promoting cooperation and progress among developers worldwide. By open-sourcing his work, Kevincoooool not only demonstrated his personal technical ability, but also reflected his contribution to promoting technological development and knowledge sharing. On this basis, we have strictly abided by the relevant open-source license agreements, made necessary references and applications to Kevincoooool's source code, and ensured that all usage behaviors comply with the provisions of the open-source agreements. This article is based on further research and exploration on top of the original work, with the aim of contributing to the knowledge accumulation and technological progress in related fields.

We once again express our most sincere thanks to Kevincoooool and user Huang Caiying. It is your openness and sharing that have provided us with valuable learning and development opportunities, and made the technology community an environment full of cooperation and innovation.

txt

MIT License

Copyright (c) 2022 Kevincoooool

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

ESP32S3 Introduction

The ESP32-S3 is a highly integrated microcontroller (SoC) launched by Espressif Systems, designed specifically for Internet of Things (IoT) applications. It is based on the Xtensa LX7 core, adopts a high-performance CPU and rich peripherals, and features low power consumption and high reliability.

Here are some of the main features and functions of the ESP32-S3:

High-performance processor: The ESP32-S3 is equipped with a high-performance Xtensa dual-core 32-bit LX7 processor with a clock frequency of up to 240 MHz, capable of handling complex computing tasks.
Memory and storage: The ESP32-S3 has 320 KB of SRAM and 8 MB of PSRAM, providing ample storage space for applications and data.
Rich peripherals: The ESP32-S3 provides a variety of peripherals and interfaces, including GPIO, SPI, I2C, UART, I2S, PWM, ADC, etc., which can be connected and communicate with various sensors, displays, and external devices.
Wi-Fi and Bluetooth: The ESP32-S3 has built-in Wi-Fi and Bluetooth functions, supports IEEE 802.11b/g/n wireless standards and Bluetooth 4.2 LE (Low Energy) protocol, and can be used to establish wireless connections with the internet and other devices.
Security: The ESP32-S3 supports various security functions, including SSL/TLS encryption, digital signatures, encrypted storage, etc., which can provide functions to protect user data and application security.
Low power consumption: Through optimized power management mechanisms and sleep modes, the ESP32-S3 runs with low power consumption, extending battery life.
Development ecosystem: The ESP32-S3 is developed based on ESP-IDF (ESP32 IoT Development Framework). Developers can use C language or MicroPython programming language for development, and can leverage rich development tools and resources.

In short, the ESP32-S3 is a powerful, highly integrated, and low-power microcontroller, suitable for various IoT applications, including smart home, industrial automation, sensor networks, and IoT gateways.

OpenMV Introduction

OpenMV IDE is a unique integrated development environment that is cross-platform, supporting Windows, Mac OS, Linux, and Raspbian operating systems.

The interface layout of OpenMV IDE is clean and easy to use. On the left are the commonly used document editing operations and code writing tools, similar to Qt Creator, with features such as code highlighting and auto-completion. The upper right area is the image area, displaying the image captured by the OpenMV lens, which can be saved and screenshotted. The lower right area is the three-color RGB histogram of the image, where feature extraction, threshold processing, and other image operations can be performed.

OpenMV IDE provides a convenient and easy-to-use development environment that meets developers' needs. Developers can develop and debug through serial connection, code execution, and other functions. In the IDE, you can click the run button to copy Python code to OpenMV, and OpenMV parses and runs it. At the same time, the IDE also provides functions such as a frame buffer viewer and histogram display, making it convenient for developers to view image data in real time and debug programs.

It also has powerful image processing functions. OpenMV IDE provides rich image processing functions, including image acquisition, preview, saving, screenshots, etc. In addition, it supports various image processing algorithms, such as feature extraction, threshold processing, edge detection, etc., which allows developers to easily implement various machine vision tasks.

Overall, OpenMV IDE is a powerful and easy-to-use development environment that provides developers with a convenient and fast machine vision development experience.

Software Preparation

OpenMV firmware source code for ESP32S3:

GitHub version: https://github.com/Kevincoooool/esp32s3_openmv_lvgl (You can support this expert, K.)

Gitee version: https://gitee.com/Reimages/esp32s3_openmv?_from=gitee_search (I found the same open-source version on domestic Gitee as the expert above. It's a clone version and is universal.)

ESP32 burning tool: flash_download_tool
OpenMV IDE: download address: https://singtown.com/openmv-download/
Virtual machine Ubuntu environment: In the Ubuntu system, install git, python3, gcc, esp-idf, micropython;

Hardware Preparation

ESP32S3 Development Board

OV2640-FPC camera (or OV5640 also works)

Purchase link: https://item.taobao.com/item.htm?_u=12t4uge5eba8&id=614238777052&spm=a1z09.2.0.0.56392e8d5FuZIw

Expansion board for connecting the development board and camera

Firmware Modification and Compilation

Use VMware Workstation Pro under Windows to create an Ubuntu virtual machine.

Virtual machine creation reference link: https://zhuanlan.zhihu.com/p/569274366

Environment Configuration

Open the terminal, and enter the following commands in the terminal to configure the environment:

Install git: password required

sudo apt-get install git

Install python3 and python2:

sudo apt install python3
sudo apt install python2.7

Install pip

sudo apt install python3-pip

Install GCC

sudo apt-get build-dep gcc

When using sudo apt-get build-dep gcc, the error "You must put some 'deb-src' URIs in your sources.list" appears.

Solution:

Enter software-properties-gtk in the terminal to open the Software & Updates window.

Check the bottom source code option, then there will be no error.

Install cmake:

sudo apt-get install cmake

Install the toolchain for ARM core compilation:

sudo apt-get install gcc-arm-none-eabi

Download esp-idf

git clone -b v4.4 --recursive https://github.com/espressif/esp-idf.git

Update esp-idf. If it crashes during execution, make sure pip is installed. Use the "pip --version" command to check whether pip is installed.

. install.sh

If you encounter python and python3 incompatibility issues, refer to: https://blog.csdn.net/Simulink_/article/details/134043709

Download micropython:

git clone --recursive https://github.com/micropython/micropython.git

Configure the esp-idf environment variables.

source export.sh

Download the ESP32S3_OpenMV firmware source code:

git clone https://github.com/Kevincoooool/esp32s3_openmv_lvgl/

If the above fails, you can also try the following method:

git clone https://gitee.com/Reimages/esp32s3_openmv

Configure the idf environment variables: execute export.sh under esp-idf. For example, my username is win, and I placed esp-idf in win. Note that there is a space between the dot and /.

. /home/win/esp-idf/export.sh

Go to the ESP32S3_OPENMV firmware directory, esp32s3_openmv_lvgl/project/ksdiy, and execute

make

If there are no compilation errors, you should see a newly created build-GENERIC_S3 folder in the project/ksdiy/ directory. We need the micropython.bin file in this directory, as well as the bootloader.bin under the /bootloader directory, and the partition_table.bin file under the /partition_table directory.

If you want to re-modify the firmware source code, you need to clear the previously compiled content first (idf.py clean), then make.

idf.py clean
make

Firmware Burning

In the project/ksdiy/ directory, you should see a newly created build-GENERIC_S3 folder. We need the micropython.bin file in this directory, as well as the bootloader.bin under the /bootloader directory, and the partition_table.bin file under the /partition_table directory.

Extract these three files to the host Windows system, and use USB or a serial connection to the (Windows) computer — either one is fine.

On the computer, open Espressif's flash_download_tool software. First erase the memory, then check bootloader.bin, partition_table.bin, and micropython.bin from top to bottom, and set the addresses to 0x0, 0x8000, and 0x10000 respectively. Click Start to burn.

Firmware based on the camera expansion board pin connection:

【Camera_expansion_board-ESP32S3_OpenMV.bin】 Download

📌 Download Center (click to jump)

📌 In Download Center -> Beginner Manual Documentation Baidu Netdisk link -> Chapter 08. [LCSC ESP32S3R8N8] Project Cases -> ESP32S3 Running OpenMV for Image Recognition.

If compilation really fails, you can use this firmware.

Firmware Testing

IDE Display

Source Code

import sensor, image, time

sensor.reset()                   # Camera
sensor.set_pixformat(sensor.JPEG)# JPEG format
sensor.set_framesize(sensor.QVGA)# 640x480
sensor.skip_frames(time = 5)
sensor.set_vflip(True)          # Set image vertical flip
clock = time.clock()            # Enable frame rate
sensor.skip_frames(time = 2000) # Delay


while(True):
    clock.tick()
    img = sensor.snapshot()
    print(clock.fps())

Verification

If the frame rate is very low, please format the development board recognized by the computer using the default format.

Line Recognition

Source Code

python

# Line detection example
#
# This example shows how to find lines in an image. For each line object
# found in the image, a line object containing the line rotation is returned.

# Note: Line detection is done using the Hough transform:
# http://en.wikipedia.org/wiki/Hough_transform
# Please read the above for more information about "theta" and "rho".

# find_lines() finds lines of infinite length. Use find_line_segments()
# to find non-infinite lines.

enable_lens_corr = False # turn on for straighter lines... Turn on to get straighter lines...
import sensor, image, time

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE) # Grayscale is faster
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time = 5)
sensor.set_vflip(True)          # Set image vertical flip
sensor.skip_frames(time = 2000)
clock = time.clock()

# All line objects have a `theta()` method to get their rotation angle.
# You can filter lines based on the rotation angle.

min_degree = 0
max_degree = 179

# All line segments have `x1()`, `y1()`, `x2()`, and `y2()` methods to get their endpoints.
# A `line()` method to get all four tuple values above, which can be used for `draw_line()`.

while(True):
    clock.tick()
    img = sensor.snapshot()
    if enable_lens_corr: img.lens_corr(1.8) # for 2.8mm lens...

    # `threshold` controls how many lines in the image are found. Only lines with
    # edge difference magnitude sums greater than `threshold` are detected...

    # `threshold` controls the lines detected from the Hough transform. Only lines greater
    # than or equal to the threshold are returned. The correct threshold value for the
    # application depends on the image. Note: the magnitude of a line is the sum of all
    # Sobel filter pixel magnitudes that make up the line.

    # `theta_margin` and `rho_margin` control the merging of similar lines. If the
    # theta and rho value differences of two lines are less than the margins, they merge.

    for l in img.find_lines(threshold = 1000, theta_margin = 25, rho_margin = 25):
        if (min_degree <= l.theta()) and (l.theta() <= max_degree):
            img.draw_line(l.line(), color = (255, 0, 0))
            # print(l)

    print("FPS %f" % clock.fps())

# About negative rho values:
# About negative rho values:
#
# A [theta+0:-rho] tuple is the same as [theta+180:+rho].
# A [theta+0:-rho] tuple is the same as [theta+180:+rho].

Verification

Grayscale Color Block Recognition

Source Code

python

# Automatic grayscale color tracking example
#
# This example shows single-color automatic grayscale color tracking using OpenMV.

import sensor, image, time
print("Letting auto algorithms run. Don't put anything in front of the camera!")

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time = 5)
sensor.set_vflip(True)          # Set image vertical flip
sensor.skip_frames(time = 2000)
#sensor.set_auto_gain(False) # Color tracking must have auto gain
#sensor.set_auto_whitebal(False) # Color tracking must turn off white balance
clock = time.clock()

# Capture the color threshold of the center of the image.
r = [(320//2)-(50//2), (240//2)-(50//2), 50, 50] # 50x50 center of QVGA.

print("Auto algorithms done. Hold the object you want to track in front of the camera in the box.")
print("MAKE SURE THE COLOR OF THE OBJECT YOU WANT TO TRACK IS FULLY ENCLOSED BY THE BOX!")
for i in range(60):
    img = sensor.snapshot()
    img.draw_rectangle(r)

print("Learning thresholds...")
threshold = [128, 128] # Middle grayscale values. Middle grayscale values.
for i in range(60):
    img = sensor.snapshot()
    hist = img.get_histogram(roi=r)
    lo = hist.get_percentile(0.01) # Get the CDF of the 1% range of the histogram (adjust as needed)!
    hi = hist.get_percentile(0.99) # Get the CDF of the 99% range of the histogram (adjust as needed)!
    # Average percentile values.
    threshold[0] = (threshold[0] + lo.value()) // 2
    threshold[1] = (threshold[1] + hi.value()) // 2
    for blob in img.find_blobs([threshold], pixels_threshold=100, area_threshold=100, merge=True, margin=10):
        img.draw_rectangle(blob.rect())
        img.draw_cross(blob.cx(), blob.cy())
        img.draw_rectangle(r)

print("Thresholds learned...")
print("Tracking colors...")

while(True):
    clock.tick()
    img = sensor.snapshot()
    for blob in img.find_blobs([threshold], pixels_threshold=100, area_threshold=100, merge=True, margin=10):
        img.draw_rectangle(blob.rect())
        img.draw_cross(blob.cx(), blob.cy())
    print(clock.fps())

Verification

QR Code Recognition

Source Code

python

import sensor, image

sensor.reset()
sensor.set_pixformat(sensor.GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # can be QVGA on M7...
sensor.skip_frames(time = 5)
sensor.set_vflip(True)          # Set image vertical flip
sensor.skip_frames(30)
#sensor.set_auto_gain(False) # must turn this off to prevent image washout...
while(True):
    img = sensor.snapshot()
    img.lens_corr(1.8) # strength of 1.8 is good for the 2.8mm lens.
    for code in img.find_qrcodes():
        print(code)

Verification

Face Recognition

Source Code

python

# Face recognition example
#
# This example shows the built-in face detection function of the OpenMV Cam.
#
# Face detection works by using a Haar Cascade feature detector on the image. A Haar cascade
# is a series of simple region contrast checks. For the built-in front-face detector, there
# are 25 stages of checks, each with hundreds of checks per block. Haar Cascades run very
# fast because later stages are only evaluated after previous stages have passed. In addition,
# your OpenMV uses a data structure called an integral image to quickly perform each region
# contrast check in constant time (the reason feature detection is grayscale only is because
# of the spatial requirements of the integral image).

import sensor, time, image

# Reset the sensor
sensor.reset()

# Sensor settings
sensor.set_contrast(3)
sensor.set_gainceiling(16)
# HQVGA and GRAYSCALE are the best for face tracking.
# HQVGA and grayscale are best for face recognition
sensor.set_framesize(sensor.HQVGA)
sensor.set_pixformat(sensor.GRAYSCALE)
# Note that face recognition can only use grayscale images
sensor.skip_frames(time = 5)
sensor.set_vflip(True)          # Set image vertical flip

# Load the Haar operator
# By default, this will use all stages. Lower stages are faster but less accurate.
face_cascade = image.HaarCascade("frontalface", stages=25)
# image.HaarCascade(path, stages=Auto) loads a haar model. The haar model is a binary file.
# If this model is custom, the quotes contain the path to the model file; you can also use
# the built-in haar model, such as "frontalface" face model or "eye" eye model.
# When the stages value is not passed in, the default stages are used. Setting the stages
# value smaller can speed up matching, but will reduce accuracy.
print(face_cascade)

# FPS clock
clock = time.clock()

while (True):
    clock.tick()

    # Take a photo
    img = sensor.snapshot()

    # Find objects.
    # Note: Lower scale factor scales-down the image more and detects smaller objects.
    # Higher threshold results in a higher detection rate, with more false positives.
    objects = img.find_features(face_cascade, threshold=0.75, scale=1.35)
    # image.find_features(cascade, threshold=0.5, scale=1.5), the larger the thresholds,
    # the faster the matching speed, but the error rate will also increase. scale can
    # scale the size of the matched features.

    # Draw a box on the found target to mark it
    for r in objects:
        img.draw_rectangle(r)

    # Print FPS.
    # Note: The actual FPS is higher; streaming FB makes it slower.
    print(clock.fps())

ESP-IDF Beginner Guide

Arduino Beginner Guide

MicroPython Beginner Guide

Module Porting Manual [Based on IDF]

Display

Sensor

Wireless Communication

Control

Project Cases

Training Camp Activities

ESP32S3 Introduction

OpenMV Introduction

Hardware Preparation

Firmware Modification and Compilation

Environment Configuration

Firmware Burning

Firmware Testing

IDE Display

Source Code

Verification

Line Recognition

Grayscale Color Block Recognition

QR Code Recognition

Face Recognition

​

ESP32S3 Introduction ​

OpenMV Introduction ​

Hardware Preparation ​

Firmware Modification and Compilation ​

Environment Configuration ​

Firmware Burning ​

Firmware Testing ​

IDE Display ​

Source Code ​

Verification ​

Line Recognition ​

Grayscale Color Block Recognition ​

QR Code Recognition ​

Face Recognition ​

ESP32S3 Introduction

OpenMV Introduction

Hardware Preparation

Firmware Modification and Compilation

Environment Configuration

Firmware Burning

Firmware Testing

IDE Display

Source Code

Verification

Line Recognition

Grayscale Color Block Recognition

QR Code Recognition

Face Recognition