Checking a running process in WebODM
reposted from smathermather.com
Edit: the dreaded “Reconstructing all views” message has been replaced with a progress monitor! But, how to dig into the back end and explore the machine that does the work is always a helpful skill to have… .
- We’ll learn how to check how far along a process is when it is calculating depthmaps and giving no feedback
- Along the way, we’ll also learn how to look at the docker images running for WebODM and
- login to said docker images in order to
- inspect that status of processing data.
- Let’s go!
So, you threw a really big dataset at WebODM, and now you are waiting. It’s been hours, maybe days, and it’s stuck on the dreaded “Reconstructing all views”:
Did you make the depthmap resolution really high, because you wanted really detailed data? Did you make it too high? How long is this going to take?
I have had this dilemma too. Sometimes I just get disgusted with myself and my predilection for choosing ridiculously computationally expensive settings, kill the process, turn the settings down a bit, and restart. But this can waste hours or days of processing, which feels wrong.
We could do the alternative. We could poke under the hood of WebODM and see how it’s progressing. For this project that’s been running for 146+ hours, this is what I decided to do.
The depthmaps that are running are being done in MVE, which can give us really detailed information about progress, but unfortunately, it makes logs a real mess, so we have it logging nothing. Let’s see how we can get around this and check in our status.
Finding the docker instance and logging in:
First, we log into the machine where WebODM is running. We need a list of the existing docker machines, as we need access to the correct machine to look at how things are progressing.
The result should give us something like this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4b0659fe6761 opendronemap/webodm_webapp "/bin/bash -c 'chmod…" 38 hours ago Up 38 hours 0.0.0.0:443->8000/tcp, 0.0.0.0:80->8080/tcp webapp 0e26ebf918f2 opendronemap/webodm_webapp "/bin/bash -c '/webo…" 38 hours ago Up 38 hours worker 1954c5136d44 redis "docker-entrypoint.s…" 38 hours ago Up 38 hours 6379/tcp broker bdc69502ca50 opendronemap/webodm_db "docker-entrypoint.s…" 38 hours ago Up 38 hours 0.0.0.0:32769->5432/tcp db 81f401a0e138 opendronemap/nodeodm "/usr/bin/nodejs /va…" 38 hours ago Up 38 hours 0.0.0.0:3000->3000/tcp webodm_node-odm_1
We want to access the webodm_node-odm_1 node, in most cases. To do this we use docker exec as follows:
docker exec -it webodm_node-odm_1 bash [email protected]:/var/www#
Woah! We are now inside the node!
Checking the available data directories:
Typically, if we only have one process running, there will only be one dataset in the /var/www/data directory
Checking our depthmap data from MVE:
For depthmaps nearly complete in MVE, there will be a file called depth-L1.mvei. We need to find out how many these are as compared with the number of images that we need depthmaps for. We’ll use a combination of the find command and wc (or word count):
find . -name depth-L1.mvei | wc -l
In my case, I have 2,485 images, or roughly 2/3s of my images processed. Looks like I am 6 days into a 9 day process before we get done with the MVE step.
I guess I will wait until Monday to check again… .5