Map the Paths Image / Video Processor and Uploader

Hi everyone!

Trek View makes maps of the natural world and uses computer vision to help better understand it (health of trees, classification of trees, accessibility of footpaths, etc).

Mapillary, Google Street View, etc. are useful tools to promote our work. But like every bit of software, there’s always short comings for individual use-cases, including ours.

Hence, we built a custom uploader and sequence manager for our captures (though it will work for any type of outdoor map captures).

In short it,

  • Adds context to manage your sequences (give it a title, description, transport type used to capture)
  • Performs local object detection’s (to automatically tag images for search)
  • Allow you to use video files for upload (will extract GPS telemetry and split video into jpg frames)
  • Lets you modify / add GPS track in case of no telemetry (useful for when using high precision GPS receiver)
  • Modify heading / pitch information
  • Add custom nadir (if image = equirectangular)
  • And finally, uploads to destinations (Mapillary, GSV…)

It’s modularised (e.g for adding new cameras / data standards / upload destinations) and all open-source too (for any devs out there).

Want to know more? Here’s some mock ups we used to design the app:

If you want to test an early version or have feedback about what’s missing from the uploader after reading the docs, let me know here.

Thanks!

1 Like

Comments on the Open Path Map Uploader UI document, copied from my email 23 July.

Hello Dave,

Slide 2

  • As for phase 2, I think doing all the processing in the cloud would be quite a load.

Slide 7

  • I miss GSV CaMM processing.

  • There is an Intel library for object recognition for free.

  • Best is that video frames and .jpg’s are not leveled by the camera, as long as GSV does not accept levelled video frames. Video frames won’t be levelled anyway. With a fixed vehicle mount, images will appear levelled, but for backpack and helmet mount all will be tilted. Cameras should provide rotation vector and/or Gpano pitch and roll XMP tags. These can be used to level the images.

  • I think it is legacy that sensors and i/o systems were not fast enough to do photo capture at 7 fps. The Mapillary smart phone app does 4K capture at a 3 meter interval at 80 kmph, which is about 7 fps. 8K may be a bit more of a challenge, but I feel Labpano cameras can do that. My point is that I feel that jpg should be the default, not video. As long as GSV needs its GSV video input, then the challenge is the other way around, create GSV video from jpg’s. In general, we want ‘capture once, use multiple’.

Slide 9

  • What I see about the UX gives me the impression that the presumption is that a complete tour is submitted in one go. However, to my experience, captures rarely cover a defined tour, if the stretch can be defined as a tour at all. E.g. Slow Ways can be used to walk one’s own tour, but Slow Ways itself is only a network of connected stretches, not tours. It seems to me that in the Uploader you need to drop the concept of Tour and better use the concept of Contribution, Batch, Capture or Sequence.

Slide 10

  • Referring to 9, I think better define the hierarchical structure of capture Session with start date, start time, start location, name (optional) and its Captures. Before mentioned properties can be get from meta-data in (first) capture with the possibility to edit them.

Slide 14

  • Labpano cameras are missing. They have the easiest GSV video workflow and are in the prosumer price bracket. I also think that you should strive for the sweetspot of 8K resolution.

Slide 22

  • I like the idea of outlier smoothing. Better than the GSV snapping. However, we better have cameras with high-precision GNSS and INU/IMU with dead reckoning to have high-precision location determination and continuous availability of that data when passing under bridges or through tunnels. Refer https://www.u-blox.com/en/product/zed-f9k-module

ZED-F9K module u-blox

ZED-F9K module . High precision dead reckoning with integrated IMU sensors

. That will help CREATE proper maps instead of enriching them with 360 imagery.

Slide 23

  • Consult Dean Zwikel for an alternative approach. (In c.c.)

Slide 24

  • Automatic tagging on the Session and Capture level, do not seem appropriate to me, manual could be. Automatic tagging on frame level could be.

Slide 25

  • I would rather use the term ‘upload channel’.

Slide 27

  • Three fixed sizes for a nadir cap is horrible, have a slider to adjust its size.

Slide 28

  • Allow user selection of output directory.

That’s what popped to my mind for the moment.

:clap: :clap: :clap:

Thanks @OmniSynThesis360VR for taking the time to renew.

I won’t tackle every point here, but will address some of the key ones:

Slide 27

  • Three fixed sizes for a nadir cap is horrible, have a slider to adjust its size.

Great idea. Will implement this. Maybe between 5% and 20%?

Slide 22

  • I like the idea of outlier smoothing. Better than the GSV snapping. However, we better have cameras with high-precision GNSS and INU/IMU with dead reckoning to have high-precision location determination and continuous availability of that data when passing under bridges or through tunnels.

We’re going to leave this up to the user. Both GSV and Mapillary modify GPS once uploaded anyway. If user wants to add more accurate path, they can use the GPS track upload function.

Slide 14

  • Labpano cameras are missing. They have the easiest GSV video workflow and are in the prosumer price bracket. I also think that you should strive for the sweetspot of 8K resolution.

The camera modules will be modularised. We tried contacting LabPano, but no response. Maybe you have an in @OmniSynThesis360VR?

Slide 9

  • What I see about the UX gives me the impression that the presumption is that a complete tour is submitted in one go. However, to my experience, captures rarely cover a defined tour, if the stretch can be defined as a tour at all.

In the uploader, we just deal with sequences. The workload will be Process image (desktop uploader) > Send to Map the Paths (web app) > Choose integrations to share to (GSV, Mapillary, Facebook, OpenTrailView).

Most of the metadata (e.g transport) is only usable by Map the Paths (there will be a search engine so users can write queries like; show me all tours captured using an electric car). You’ll be pleased to know we also plan to add a tour function (user can group and order their sequences together).

1 Like

Which cameras provide this information?
My iPhone has a water compass app, so my iPhone could record that data, but I haven’t found an app, that records gps data and records pitch (and tilt/yaw) (fixed to a relative position of the camera, should work!?)

Why? Because that is done by software?

In the slides I saw ‘panellum’ mentioned, PSV (photo sphere viewer) it has better capabilities in my opinion, I’ve convinced Nick, he is starting to use it :wink:
(Also, I have created a POC with PSV that enables anyone to place geo referenced tags in 360 images!)

Question, I do not have the capability to level out images nicely with software, not automated Anyway. Thus I bought a moza guru 360 to get a better bases… for more general purpose, could it be possible to tick a checkbox ‘level image’?

Last one, could a generic publishing tool be created? Add a name/password or a hash string and a target in your tool, and provide a tool one could implement on His or her own hosting account?

1 Like

@dgreenwood-trekview you are welcome.

Slide 27
GSV has a max as to how big a nadir cap may be. Suggest that you max out on that so that users do not run into issues with GSV processing.

Slide 22
Eh, we better be precise in language. GSV and Mapillary do not modify GPS, they modify location. Imprecise and ambivalent wording always leads to issues in information systems.
You are suggesting that uploading a location track would result in a more accurate path. Don’t get this, as this should do no better than a proper camera for the job.

Slide 14
In the email correspondence, I have introduced Anna from Labpano to you already.

Slide 9
Then I do not understand the hassle of naming sequences.
Guess you mean ‘workflow’ instead of ‘workload’.
‘Transport’ as an example of meta-data? I have no clue as to what you are referring.
Will the search engine search in tours, sequences or both?
With only the slides as a reference to what you are up to, it seems to me that the terms tour and sequence are mixed up at times.

Hoi Eesger, you are right in questioning. I feel the whole matter might turn out to be a moving target. My aim is to capture once and use multiple. An issue is that Google is not that open in its roadmap.
Any GSV certified camera is supposed to provide that info, but it seems the certification process is not that rigid.

Your iPhone has a gyroscope and it has an accelerometer, the basic sensors for an Inertial Navigation Unit. If you capture a pano with your iPhone using the GSV app, then the necessary Gpano XMP tags will be in the .jpg file.

I guess that GSV doesn’t want the video frames levelled, because than the flow of the camera movement is disrupted and cannot be used for analysis anymore. But what they are doing today, several years after these specs have been set, that I do not know. Officially, GSV video capture is bêta still. GSV processing does level the imagery. It seems to be done by image analysis, which of course works quite well in a human created (city) environment, but is flawed in nature.

As Qualcomm integrates GNSS receiver and INU in its System on a Chip (SoC) and a manufacturer as U-Blox has integrated modules too, even with multi-channel, multi-GNSS reception, doing on-device dead reckoning, then location precision up to the decimeter can be achieved, even in troublesome environments and lacking GNSS signal for some time, as the Mosaic 51 features. Then the choice is there which processing and calculation one does where, the cloud or on its edge. I would choose the latter. High-precision also means that GSV and Mapillary should not do ‘snapping’ of sorts to map defined roads and trails, but that the map should be improved based on the location of the imagery, unless the map is based on professional surveying data.

Labpano’s cameras have GNSS receiver and INU. So, the meta-data is available. Although they have GSV certification, their meta-data in the GSV video CaMM track and .jpg files are not according spec. This is on the improvement roadmap. @JimGayes360 and I have also asked to implement 1, 3, 7 fps interval capture with jpg output. It is a matter of capacity and priorities.

Hi @OmniSynThesis360VR! (you wrote ‘hoi’, you are Dutch?)

I now use the ‘GPX tracker’ what I’dd love is that such an app would also record tilt/yaw/pitch… then I could use that data to improve the orientation of my imagery…

I’m not that big a Google fan (to big, and they already know to much…) I’dd love to have a more generic solution…

@Eesger Meanwhile I have added to my previous.

Yes, first 50 years in NL, but after all, still in the kingdom :wink:

To be honest I am not happy with both, poor support. One gets serious attention only, if one has a mass contribution to their aim, i.e. rather opportunistic. Understand your reasoning too. Mapillary used to be that one, independent of big tech, but it that has changed now that is in the assimilation process with Facebook.

What camera are you using?

I now use a YI360 (more info here, to prevent going off topic from this thread, reply there?)

Hi all,

A quick update on this: we’re into the final stages of testing for release.

You can take a look at some of the features here:

In short you can:

  • Add metadata to sequence for easy management and discovery (name, description, tags)
  • Convert video to photo frames
  • Geotag images using gpx track
  • Define photo spacing
  • Modify photo GPS information (position and heading)
  • Add a nadir
  • Blur people and license plates
  • Upload to web platforms
    • Map the Paths Web
    • Mapillary
    • Google Street View
    • Developer? Add your own…

If anyone would like a pre-release version, let me know, and I’ll get one over to you on two conditions:

  1. You accept it might not work perfectly
  2. You are willing to provide feedback on where it is not working perfectly

Do I read correctly that you calculate GPS position in the resolution second?

Sorry when I’m blunt, but this isn’t good enough… When you drive, or even cycle your speed is at least 5 meters per second!

When you have a ffmpeg sequence you have sub-second data, do not discard that added resolution!!

First, above all, the faster you go the more important it is to sync your GPX recorder time and your camera time! Most devices have an option to sync with a time server…

This is what I do:

  • read gpx into database
  • allow sub second time offset of track (find a keypoint at a corner and sync with track)

After extracting the images from the mpg, for each image do:

  • find the closest matching time
  • find the next match in the GPX sequence (if exists)
  • calculate the average GPX position, weighted by difference in time
  • on exact time match img > GPX:
    • find the previous match in the GPX sequence (if exists)
    • calculate the average GPX position of the next and previous and average with exact match

This gives a weighted GPX position which is (almost) always based upon on two or three GPX recordings. resulting in a more stable position and a smoother track.

The results are quit good with this approach!!

2 Likes

AD: up until now I have done most of my tracks walking, thus my sequences have been recorded at one image per second.

Now I add GPS data to imagery and remove images that are taken within 1.5 meters. I now have several cycling sequences I need to process. I have recorded those sequences in 2 images per second. Therefor I will have two images with EXIF data in the same second. I will add a function where store the frame rate (EXIF FrameRate should be possible) and add that half a second every other image. This will increase the quality of the GPX calculations!

PS: I’m extremely busy with my regular work now, so I haven’t gotten around to that yet :wink:

1 Like

Thanks @Eesger

I use exiftool right now (for extracting gps from video file), which is not perfect (and is the reason we have simple 1 second resolution). This was chosen because I found it hard to extract encoded telemetry (CAMM, GPMF) using ffmpeg.

Can you give me an example of how you extract telemetry using ffmpeg?

Good question, I will need to dive into the code, when I find the time, I’ll tell you more

1 Like

@Eesger, I think Google recommends 3 fps for cycling.

HQ | Blog | Spotted a Trekker? | Become a Trekker | Facebook | Instagram