ODM Benchmarks
This repository is a part of the ODM project and is dedicated to benchmark sample datasets found in the ODMData repository and a few other thrid party sources, to better understand the behavioural characterists of ODM.
OpenDroneMap benchmarking data is summarized in the following pages:
- View benchmark data organized by dataset (TXT)
- View benchmark data organized by ODM version (TXT)
- View benchmark raw data (CSV)
Datasets for Benchmarking
Preferred datasets for benchmarking at this time are:
- Brighton Beach (18 photos)
- Toledo (87 photos)
- Wietrznia (225 photos)
- Shitan (493 photos)
These and many other sample datasets are indexed at the ODMData repository.
How to Contribute Benchmarks
This section provides instructions for contributing to the ODM Benchmarks project.
-
Review Previous Benchmarks – You can have a look at previous benchmarks which is found in the
benchmarks.csvfile, located in thedatadirectory to understand various things such as for example, figuring out which dataset has been benchmarked lesser than in comparision to other datasets. -
Select a desired Dataset – Choose one of the datasets of your choice which is listed in the table seen above. Links to download the dataset is provided in the same.
-
Run the Dataset – Create a new task in ODM and process the images.
-
Verify the results – Ensure that the task completed successfully by viewing the maps and models generated
-
Submit Your Results – Open the
benchmarks.csvfile and look at the header for each of the column and note the desired information required before submitting and edit the csv file. Submissions are accepted Via a Pull Request or you can choose to post your results on the ODM community forum. A table is given below to explain the structure of the CSV.
| Attribute | Explanation | Example |
|---|---|---|
| ID | Benchmark Number | 1 |
| DATASET | Dataset Name | Toledo |
| PROCESSING_TIME | Time taken to Process The dataset | 1h 9m |
| PROCESSING_SUCCESS | Confirm If process was successful | Y |
| ERROR_TYPE | Mention error if any occured | – |
| RAM_SIZE | Size of RAM allocated | 16 GB |
| RAM_CLOCK_SPEED | RAM Frequency of the Machine | 2133 MT/s |
| CPU_TYPE | Make and model of the CPU | Intel i5 |
| CPU_CLOCK_SPEED | Clock Speed of the CPU | 2.3 Ghz |
| CPU_NUM_CORES | Number of Cores of the CPU | 4 |
| STORAGE_TYPE | Storage Type of the system. HDD/SSD/NVME | SSD |
| OS | Operating System of the Machine | Ubuntu 18.04 |
| VM_TYPE | Virtual Machine on which ODM is running on | Docker |
| ODM_VERSION | Version of ODM Used to Benchmark | 1.3.1 |
| ODM_CLUSTER | Delcaration of usage of Cluster ODM | N |
| CONFIG_NAME | Mention of the Configuration in which the dataset is being processed | Default |
| CONFIG_RESIZE | mention of image size if images are being resized | 2048 px |
| CONFIG_OTHER | Other Configurations made that are worth mentioning | – |
| TEST_DATE | Date of Test | 2020-03-07 |
| TEST_BY | Information of the individual who tests the data | Corey Snipes |
| INCLUDE_IN_SUMMARY | Check if Data has been included in summary | Y |
| NOTES | Additional Notes | – |
Queries and Contributions
-
Questions and Doubts – Any queries you may have can be posted in the ODM community forum.
-
Contributing new Benchmark Data – If you have benchmark data to share, See this page for details on contributing.