5. Xprize Machine Learning (ML) Classifiers

Machine learning (ML) is the subset of artificial intelligence (AI) that focuses on building systems that learn—or improve performance—based on the data they consume.

For the Xprize, we shall rely on some Machine Learning Classifiers to automatically assign various images and sound recordings to their appropriate species names in the animal kingdom.

Before you start running any ML Classifiers, ensure that Docker Desktop is up and running. In case you are using Rancher-Desktop, ensure it is running before executing code. Running these ML Classifiers requires that either Docker or Rancher-Desktop is running, depending on the one you’re using.

Downloading the Machine Learning Classifier Files

Your supervisor will provide the links to the folders containing the Machine Learning (ML) classifiers. However, if needed, here is the link.

Download and extract your ML folders to the xprize folder we created in Chapter 2 using WSL2 and from within which we called VS Code using the command code ..

Here is a step by step procedure of doing this.

  1. If the supervisor has already provided the Teams access, you can go to the Teams Folder containing the ML folders, and click the three ellipsis after the Sync button. Click on Download.

  2. Team will store the ML Models folders as a zipped folder inside your Downloads directory or the default directory for your file downloads.

  3. Extract the zipped folder. Copy the contents of the extracted zipped folder to the Ubuntu folder called xprize which we have created in Chapter 2. You can go back to that chapter and follow the instructions.

Below is when we are copying the ML folders from the downloads directory (the zipped folder was renamed to xprize_ml_models for easy identification).

The extracted ML Folders

Here are the extracted ML folders copied to the xprize folder within our Ubuntu environment.

Folders saved within Ubuntu

As a reminder you can open VS Code from within a Linux/Ubuntu environment using the following procedures:

  1. Open WSL2 from Start.

  2. List all your folders from within your Ubuntu home/<username>/ directory using the ls command.

  3. Get into the xprize directory using cd xprize/ command.

  4. Open VS Code from within xprize folder using this command –code ..

And here they are as they appear within VS Code.

The Xprize Folders in VS Code

Creating a Virtual Environment

What is a Virtual Environment?

A virtual environment is a tool that helps to keep dependencies required by different projects separate by creating self-encapsulated setting which is free from interference by any other project.

A virtual environment is useful when one is working with various dependencies across various projects. For example, differing version numbers can cause conflicts across projects. A virtual environment keeps dependencies used in each project in their own isolated space.

To create a virtual environment, run the following code:

python3 -m venv .venv

where venv stands for virtual environment and .venv is the name of our virtual environment.

Notice that a new folder called .venv appears at the top.

Now that we have created a virtual environment, let’s activate it. Run this:

source .venv/bin/activate

We are using the keyword source because we are within a Linux environment and using a bash terminal. If it was a regular python environment in a Windows Powershell terminal our starting command would have been py.

Nevertheless, let’s continue.

Your namespace should now have the (.venv) in brackets appended to the cursor name.

If you’ve reached here, so far so good.

Virtual Environment Activate

Now that we have our virtual environment activated, we can start running our models. For tutorial purposes, we shall first start by pulling all the models. Secondly, we can thereafter run them sequentially.

Installing the ML Classifiers

Since all classifiers are installed at the project level, that is, within the xprize folder rather than within the respective model subfolder such as birdnet_amphReptiles, we shall perform the installation of the ML Classifiers first.

So within the xprize folder, we will start by installing the following classifiers in order.

  1. birdnet_amphReptiles

  2. birdnet_birds

  3. birdnet_mammals

  4. megadetector_species

birdnet_amphReptiles Classifier

To install this model, run the following code from within the xprize folder.

docker pull naturalstate/birdnet_amph_reptiles:latest

You should get an output like below. However, mine is shorter since I already had the model pre-installed. However, if installing for the first time, the terminal output will be much longer.

Installing an ML Classifier

birdnet_birds Classifier

To install this model, run the following code from within the xprize folder.

docker pull naturalstate/birdnet_birds:latest

birdnet_mammals Classifier

To install this model, run the following code from within the xprize folder.

docker pull naturalstate/birdnet_mammals:latest

megadetector_species Classifier

To install this model, run the following code from within the xprize folder.

docker pull naturalstate/md_vit:latest

Troubleshooting

Sometimes, due to some mispelling or the model being unavailable, you may encounter the following error:

Error response from daemon: pull access denied for naturalstate/mammals, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

One way could be you try to login to docker using the docker login -u <username> and your Personal Access Token (PAT). Secondly, try running the docker pull command again. If the error shows up, the model is unavailable or its docker hub name has changed.

The following error may occur when keying in the docker run command.

usage: main.py [-h] --phase_nb PHASE_NB --acoustic_test_data
               ACOUSTIC_TEST_DATA [--tmp_prediction TMP_PREDICTION]
               [--birdnet_processed BIRDNET_PROCESSED]
               [--detections DETECTIONS] [--sensitivity SENSITIVITY]
               [--min_conf MIN_CONF] [--rtype RTYPE]
               [--remove_audio REMOVE_AUDIO] [--save_out_audio SAVE_OUT_AUDIO]
main.py: error: unrecognized arguments: --slist /app/slist

Related to the first error, this occurs when running a non-existent model. Try pulling the correct model and make tweaks where necessary, such as pwd() line to match the name of the correct model existent on docker hub.

Running the ML Classifiers

Now that we have pulled the necessary classifiers, its time to run them. By now you have noticed why we installed Docker Desktop. It is to enable us pull and run the ML Classifier images from Docker.

Just like we pulled the models sequentially, we shall run them sequentially. However, this time round, running the ML classifiers will be done at the classifier subholder rather than the external xprize folder.

Running the birdnet_amphReptiles classifier

Go to the birdnet_amphReptiles folder like below:

cd birdnet_amphReptiles/

Thereafter, run the birdnet_amphReptiles model like so:

docker run \
    --shm-size=10.24gb \
    -v $(pwd)/data:/app/input \
    -v $(pwd)/birdnet_processed:/app/birdnet_processed \
    -v $(pwd)/tmp_prediction:/app/tmp_prediction \
    -v $(pwd)/detections:/app/detections \
    -v $(pwd)/species_list:/app/slist \
    naturalstate/birdnet_amph_reptiles:latest \
    --phase_nb 0 \
    --acoustic_test_data /app/input \
    --slist /app/slist

For some laptops, and preferably so, a long list of outputs should be printed out indicating the progress of the model. For some (like mine), no output is printed out. However, the IMPORTANT thing is that the model is doing some predictions. You should see the prediction output under the detections folder. A csv file name with experiment date, time in Greenwhich Meridian Time (GMT) format and some other numerical values should be printed.

Running a ML Classifier

How the model works

The model takes a species list as provided within the species_list/species_list.txt file and assigns the appropriate species name according to the audio file (.wav). Inside the csv file, the species name is within the scientific_name column while the name of the audio file is within the parent_name column.

File containing prediction values

Thus, using our example image above, the species name of the animal associated with the audio recording 20240520_130000_3402.0_3405.0.wav is Dendropsophus triangulum. Additionally, the confidence scores for this prediction are 0.56. The threshold, which is the pass mark to start gaining confidence with the prediction, is 0.5 and its consistent across all species.

The birdnet_processed folder contains the following folders.

  1. originals - This folder contains all the audio files.

  2. Species folders - These are the folders, whose namespaces are the species names, that contain the audio files predicted for that species name. For example, in my case I have to other folders apart from the originals folder. They are Dendropsophus triangulum/ and Rhinella marina/ folder. These contain the audio files predicted for each of the respective species.

Prediction folders

Folder Orientation

There are couple of sub-folders within the birdnet_amphReptiles folder. Let’s brisk through each one of them.

birdnet_processed sub-folder

This sub-folder will contain a folder of the experiment’s date and time, which further contains other sub-folders containing the originals/ sub-folder containing all the audio files, and other sub-folders containing the predicted audio file. Each of the other sub-folders will be named by the species name and will contain the predicted audio-files associated with the species-name.

data sub-folder

This sub-folder contains the audio files that will be passed throught the ML Classifier.

detection sub-folder

This sub-folder contains the csv file holding the prediction scores, and the scientific name associated with each of the audio files found in the data sub-folder.

species_list

This sub-folder contains the species_list.txt that is used by the ML Classifiers to give a (scientific name) to every unclassified audio file.

tmp_prediction

This is a temporary storage folder for the classifiers predictions.

It is ideally empty if the ML Classifier worked. However, in case there was an error, some folders will appear here.

Running the birdnet_birds classifier

Go to the birdnet_birds folder like below:

cd birdnet_birds/

Thereafter, run the birdnet_birds model like so:

docker run \
    --shm-size=10.24gb \
    -v $(pwd)/data:/app/input \
    -v $(pwd)/birdnet_processed:/app/birdnet_processed \
    -v $(pwd)/tmp_prediction:/app/tmp_prediction \
    -v $(pwd)/detections:/app/detections \
    -v $(pwd)/species_list:/app/slist \
    naturalstate/birdnet_birds:latest \
    --phase_nb 0 \
    --acoustic_test_data /app/input \
    --slist /app/slist

For some laptops, and preferably so, a long list of outputs should be printed out indicating the progress of the model. For some (like mine), no output is printed out. However, the IMPORTANT thing is that the model is doing some predictions. You should see the prediction output under the detections folder. A csv file name with experiment date, time in Greenwhich Meridian Time (GMT) format and some other numerical values should be printed.

How the model works (to clarify)

The model goes through the data folder and classifies which audio files belong to birds. The prediction scores of the output of the audio files are found within the csv file created in the detections folder. For some reason only one row is present in the predictions csv file.

Folder Orientation

birdnet_processed sub-folder

This subfolder holds the experiment folder whose namespace contains current date and time (GMT format) which contains the following sub-folders with 1) originals referring to the original audio files (in this case only 1 file) and, 2) other folders holding the audio file classifications with the name of the folder in this case, such as subthreshold referring to the classification of the audio file.

data sub-folder

This contains the experiment date and time subfolder which holds all the audio files to be classified.

detections sub-folder

This sub-folder contains the csv file holding the prediction scores for the audio files that were classified. Only one was classified in my case.

species_list

This sub-folder contains the species_list.txt that is used by the ML Classifiers to give a (scientific name) to every unclassified audio file.

tmp_prediction

This is a temporary storage folder for the classifiers predictions.

It is ideally empty if the ML Classifier worked. However, in case there was an error, some folders will appear here.

Running the birdnet_mammals classifier

Go to the birdnet_mammals folder like below:

cd birdnet_mammals/

Thereafter, run the birdnet_mammals model like so:

docker run \
    --shm-size=10.24gb \
    -v $(pwd)/data:/app/input \
    -v $(pwd)/birdnet_processed:/app/birdnet_processed \
    -v $(pwd)/tmp_prediction:/app/tmp_prediction \
    -v $(pwd)/detections:/app/detections \
    -v $(pwd)/species_list:/app/slist \
    naturalstate/birdnet_mammals:latest \
    --phase_nb 0 \
    --acoustic_test_data /app/input \
    --slist /app/slist

For some laptops, and preferably so, a long list of outputs should be printed out indicating the progress of the model. For some (like mine), no output is printed out. However, the IMPORTANT thing is that the model is doing some predictions. You should see the prediction output under the detections folder. A csv file name with experiment date, time in Greenwhich Meridian Time (GMT) format and some other numerical values should be printed.

How the model works

The model goes through the data folder and classifies which audio files belong to birds. The prediction scores of the output of the audio files are found within the csv file created in the detections folder.

Folder Orientation

birdnet_processed/ sub-folder

Within the experiment sub-folder, there contain other sub-folders named according to the species names. The species names are found within the species list text file in species_list/ folder. These folders named with species names contain the audio files which the ML Classification has predicted belong to that species (thus the folder has the species names).

The originals sub-folder within this directory contains all the audio files found in the data sub-folder.

data sub-folder

This sub-folder contains all the audio files to be classified.

detections sub-folder

This sub-folder contains the csv file holding the predictions for each audio file.

NB If an audio file has the scientific name of subthreshold it means the ML Classifier was not able to classify the audio file. Perhaps due to the nature of the soundings or there was noise.

species_list

This sub-folder contains the species_list.txt that is used by the ML Classifiers to give a (scientific name) to every unclassified audio file.

tmp_prediction

This is a temporary storage folder for the classifiers predictions.

It is ideally empty if the ML Classifier worked. However, in case there was an error, some folders will appear here.

Running the megadetector_species classifier

Go to the megadetector_species folder like below:

cd megadetector_species/

Thereafter, run the megadetector_species model like so:

docker run \
    --shm-size=10.24gb \
    -v $(pwd)/data:/app/input \
    -v $(pwd)/md_processed:/app/md_processed \
    -v $(pwd)/test_results:/app/output \
    naturalstate/md_vit:latest \
    --phase_nb 1234 \
    --imgs_dir /app/input \
    --output_dir /app/output \
    --conf_threshold 0.75

For some laptops, and preferably so, a long list of outputs should be printed out indicating the progress of the model. For some (like mine), no output is printed out. However, the IMPORTANT thing is that the model is doing some predictions. You should see the prediction output under the detections folder. A csv file name with experiment_<date>_<time> namespace in Greenwhich Meridian Time (GMT) format and some other numerical values should be printed.

How the model works

The model works by iterating through every image found within the data/<experiment-name-subfolder> and creating two csv files. One is the experiment date and time csv within the test_results/ sub-folder. The other is the predictions.csv file found within the md_processed/<experiment-name> sub-folder. The former contains the category name of each image, where each image is categorized as an animal, subthreshold, vehicle and person.

Classification categories

The latter will match every image with its predicted species name. Classification species

Folder Orientation

data sub-folder

This folder contains the images to be classified.

md_processed sub-folder

This sub-folder contains another additional folder within it, the experiment date and time folder.

Within the experiment date and time subfolder, there are the following sub-folders:

  1. animal - this subfolder contains all the images which have been assigned the animal category by the ML Classifier.

  2. blank - this subfolder contains all the images which do not have a feature within them, say a totally black image.

  3. originals - this subfolder contains all the images found within the data folder.

  4. person - this subfolder contains all the images which the ML Classifier predicts as containing a human being in them.

  5. subthreshold - this subfolder contains images which the ML Classifier could not adequately place into any of the six categories so far.

  6. vehicle - this sub-folder contains images which the ML Classifier has labelled as vehicles.

Within the md_processed subfolder, there is an additional file called predictions.csv. This file contains the prediction scores and classification names for each image.

test_results sub-folder

This subfolder contains the experiment date and time csv file. This experiment_<date>_<time> file contains the classification of each image to one of the six major categories of animal, blank, originals, person, subthreshold, and vehicle.

The Megadetector Species directory