Processing for non-standard RS layers

In general the layers below are not available on Google Earth Engine and therefore have to be downloaded manually. As a result they each need slightly different processing in order to prepare them for upload to AGOL. Below you’ll find links to the individual scripts for each layer as well as instructions for how to acquire the raw data.

RS_008 to RS_011

These four topographic layers (TRI, TPI, TWI, and Roughness) are derived from the DEM extracted from GEE (RS_005) and are calculated in R.

NB: before continuing make sure that you have run the processing steps (01_create_gdbs.py and 02_ee_to_gdb.py) for RS_005 which is the elevation layer. It is used in the subsequent steps.

Next, create an R project in the agol-data-workflows directory. Then run the 01_topographic_processing.R script.

Package installation and versioning in the R project is dealt with using {renv}. If you aren’t already using {renv} for collaboration on R projects then it’s advisable that you look into it. See the {renv} documentation here to get started.

When you clone/fork this repo make sure you have the renv.lock file, which you can use to install the packages that were used during the development of this R code by running renv::restore(). Whitebox R package, which is used to create the topographic wetness index, is built on WhiteBox Tools which needs to be installed after the package is loaded for the first time. Follow prompts for installation after loading package.

You will then need to specify the name of the AOI used to extract GEE data which is stored in the Boundaries.gdb created in earlier steps. On execution of the code the four layers will be written to the api_input folder and are ready for upload onto AGOL.

However, you will need to run the RS_008_to_011.py script afterwards to import those layers into the Topographic.gdb.

RS_021

This is the rivers layer and it comes from the GIRES (Global Intermittent Rivers and Ephemeral Streams) database.

The first thing to do is to navigate to this repository and download the GIRES_v10_gdb.zip file which is roughly 1.5GB. Find a suitable directory on your machine and unzip the file so that you can access the GDB.

Then go to the RS_021.py script and set the gires_gdb_dir variable to the path of where you’ve stored the GIRES GDB.

The rest of the code will clip the rivers layer to the AOI, write it to the local river GDB and prepare the layer for import into AGOL.

RS_036

This is the Global Human Modification Index layer. To download the raw data go to this repository and download the gHMv1_300m_2017_static.7z file. Find a suitable directory on your machine and unzip the file so that you can access the TIFF file called gHMv1_300m_2017_static-0000046592-0000046592.tif (there are several TIFFs in the folder, each covering a different geographic area. You’ll have to load each one to determine which TIFF covers your AOI. This particular file covers a big chunk of East Africa).

You can then go to the RS_036.py script and set your ghm_raw variable which points to your local copy of the raw GHM layer. Running the code after that will process the layer and prepare it for upload to AGOL.

Note that the original values in this layer ranged from 0.0 to 1.0, where 0.0 is no human modification and 1.0 is full or complete human modification. These values were represented as 32-bit floating point values, but were converted by the database developers to a 16-bit integer to reduce file size, by multiplying by 65536. Therefore in our code there is a line that divides the cell values 65536 to get them back onto the original 0 to 1 scale.

RS_037 to RS_038

These two layers come from the Global Human Settlement Layer. RS_037 refers to the GHS Settlement Model (GHS-SMOD) layer and RS_037 refers to GHS built-up surface (GHS-BUILT-S) layer.

To download the GHS-SMOD layer, go to this data repository. On the landing page you will have to select the product and epoch (there is an option for resolution but it will auto-select 1km because there isn’t another resolution available for this particular GHS layer). Select these values as shown below.

Once you’ve done that you need to click the specific tile(s). In the case of the Mount Kenya Rangelands AOI, the footprint covers multiple tiles so we have to click on both. This will initiate an automatic download of a zipped file containing the TIFF layers.

Once you’ve downloaded the layers, unzip the GHS-SMOD files and store the TIFFs in a suitable location.

The next thing to do is to download the GHS-BUILT-S which can be accessed here. Follow the same process as above (select the correct product, epoch and resolution and click on the tiles to download the data). Once you’ve done that, unzip the files and place them in the same directory as the GHS-SMOD files.

Then go to the RS_037_to_038.py script and specify your directories for the raw layers. Then continue to run the rest of the script. If there is more than one tile, the code will mosaic the layers after clipping to the extent of the AOI.

RS_041 to RS_047

These layers come from the Gridded Livestock of the World – 2010 (GLW 3) data set. From this repository we will extract data for cattle, sheep, and goats. A layer for each animal group needs to be download separately. Follow these links: cattle, sheep, and goats.

For each livestock group click on Access Dataset and then download as a zipped file, as shown below.

Then unzip and place the contents of for all livestock groups into a single folder.

Go to the RS_041_to_047.py script and set glw_raw_dir variable to point to the location of the livestock files. Run the rest of the code to prepare layers for upload to AGOL.

RS_053

This is the roads layer derived from OpenStreetMap. The first thing to do is open and run the Google Colab notebook 17_OSM_roads.ipynb. This code uses the OSMnx Python package to as an API to access OSM data.

Given that this isn’t a GEE layer there isn’t a way to write the output directly to the GEE_exports folder that houses the other RS layers. Instead you need to mount your Google Drive as a local drive within the Colab notebook runtime using the code below.

# Mount Google Drive
from google.colab import drive
ROOT = '/content/drive'
drive.mount(ROOT, force_remount=True)

Once you’ve done that you’ll be able to see and access the contents of your drive in the side panel.

The code will use the AOI as a bounding box with which to extract the road data. It will be exported as a geopackage file to the GEE_exports folder.

Once that’s done you can run the RS_053.py script. This will filter out road types we don’t need and then send the resultant file to the api_input folder making it ready for upload to AGOL.

RS_067

This is the 1km x 1km raster canopy height layer from the GEDI L3 Gridded Land Surface Metrics data set. This layer can be downloaded from the ORNL DAAC portal found here.

In order to access the data you will need to create an EarthData account. Click on the login button and follow the prompts.

Then sign into the portal before continuing.

On the landing page, scroll down to the list of data files and select the GEDI03_rh100_mean_2019108_2022019_002_03.tif file which is the latest canopy height layer.

Add the dataset and then checkout. Then click on “order these items”. After doing so you will get an email with a link to download the data.

After following the link you will be directed to the data repository. Navigate into the data directory to locate the file and download it.

Create a suitable directory to store the TIFF file. Then go to the RS_067.py script and set the gedi_raw variable to the path of where you’ve stored the GEDI TIFF file.

The rest of the code will clip the layer to the AOI, write it to the GEDI GDB and prepare the layer for import into AGOL.

RS_068

This is the global aridity index. These data need to be downloaded from Global Aridity Index and Potential Evapotranspiration (ET0) Database: Version 3, located here. Navigate to the repository and download the Global-AI_ET0_annual_v3.zip file. Unzip the contents in a suitable location. The raster layer you need to use is called ai_v3_yr.tif

Go to the RS_068.py script and set ai_raw variable to point to the location of the aridity index layer. Run the rest of the code to prepare layers for upload to AGOL.

RS_071

This is the Soil Atlas of Africa layer. This layer needs to be requested from the European Soil Data Centre.

Once you’ve filled out the request form you will be sent a link to download the soil atlas shapefile. Once you’ve downloaded and unzipped the file, go to RS_071.py and change the soils_raw variable to match your local directory. Run the rest of the code to prepare layers for upload to AGOL.

RS_073

This is the Aboveground Live Woody Biomass Density layer from GEDI. The data can be downloaded from the Global Forest Watch repository located here. Under the details list on the right panel find “Records: 280” and click on “View data table”. From there select the appropriate tiles and click on the download link of the “Mg_ha-1_download” column. This will result in the download of TIFF files with a resolution of 30m. The MKR region covers two tiles so these need to be mosaicked (see code for more details).

Create a suitable directory to store the TIFF files. Then go to the RS_073.py script and set the raw1 and raw2 variables to the path of where you’ve stored the TIFF files. Run the rest of the code to prepare layers for upload to AGOL.

RS_074

This is a vector layer representing lithology which comes from the 2015 Global Lithological Map (GLiM). The vector layer can be downloaded in geodatabase format by following this link to the repository.

Once the geodatabase has been stored locally got to the RS_074.py script and set the lithology_raw variable to point to the local file path. Proceed with running the rest of the code to prepare layers for upload to AGOL.

RS_075

This is a raster layer representing Growing degree days (GDD) and can be downloaded using this link. The data come from The Global Environmental Stratification project, see here for more details. The zip file contains 36 TIFF files. The GDD layer is labelled as var12.tif.

Once the var12.tif layer has been stored locally got to the RS_075.py script and set the gdd_raw variable to point to the local file path. Proceed with running the rest of the code to prepare layers for upload to AGOL.

RS_076

This is the World Ecological Facets Landform Classes layer and can be accessed as an ESRI Image Service here. The layer needs to be added from path in an ArcPro session. You will then need to clip the layer to your AOI and export it from there.

RS_078

The European Commission Joint Research Centre’s Global Surface Water dataset is among the most accurate representions of terrestrial surface waters because of its relatively high resolution (30 m) and its emphasis on temporal variability. The main product of interest is the occurence dataset which represents the percentage of months from 1984 to 2021 when each LandSat pixel contained water and reflects both inter- and intra-annual variability. In general, values of 95 or higher (i.e., 95% water occurrence) can be classified as permanent water.

Data can be downloaded as individual 10°x10° tiles or larger extents can be downloaded using python scripts available from the source website. Download the relevant tiles to a local directory. Then go to the RS_078.py script and update input paths and the number of tiles to be uploaded. Run the rest of the code to prepare layers for upload to AGOL.