Fix scene flow ground truth generation in Argoverse 2

[Kin-Zhang]

As HiMo: High-Speed Objects Motion Compensation in Point Cloud highlights: LiDAR point clouds often contain motion-induced distortions, degrading the accuracy of object appearances in the captured data.

📺 Check this 40s video clip for a visual explanation:
https://youtube.com/clip/UgkxXpEk6ef0nFTTDF7ikTtwae3dMfT0ycyl?si=xqWrntJiP4bThsd4

🛠️ What's Improved?

Previously, Argoverse 2 generated scene flow ground truth by expanding bounding boxes with a fixed 20cm value. However, following HiMo, we now expand bounding boxes based on object relative velocity, leading to more accurate motion compensation.

🔍 Before & After Visualization

Below is a comparison showing the improved bounding box expansion using HiMo:

^{Visualization from demo/val, scene id: 25e5c600-36fe-3245-9cc0-40ef91620c22}

[Kin-Zhang]

📊 Impact on Evaluation Metrics

A key issue with the previous ground truth processing was inflated scores due to inaccurate bounding box annotation. Since the bounding box expansion was fixed rather than velocity-based, static and dynamic objects were not correctly distinguished, resulting in artificially lower error rates (better scores) before the fix.

The updated table below demonstrates the effect of HiMo correction. Notice how all methods show higher error values after the fix, indicating that the previous ground truth underestimated real-world motion errors (all these three pre-trained weights can be downloaded from huggingface/here:

The raw terminal result screenshot:

🔬 Summary

• More accurate motion compensation: Bounding box expansion now accounts for object velocity instead of a fixed offset.
• Improved accurate score on metrics: More reliable scene flow ground truth for dynamic objects. Fixing score inflation from previous ground truth processing.

[Kin-Zhang]

@ikhatri @kylevedder I think you may interested as we discussed before during ECCV'24. Maybe it could be also updated in the leaderboard gt results. (Although it's mainly on high-speed objects.)

[Kin-Zhang]

A further evaluation fix on some evaluation masks provided by the official mask zip file includes ground points (AV2 dataset only).
So, in #36, we perform double-mask validation to exclude ground points as well.

                if 'eval_mask' in f[key]:
-                    data_dict['eval_mask'] = f[key]['eval_mask'][:]
-                    raw_eval = f[key]['eval_mask'][:]
+                    raw_ground = f[key]['ground_mask'][:]
+                    # NOTE(Qingwen): performance might be changed for av2 since some eval_mask provided by av2 didn't remove ground points.
+                    data_dict['eval_mask'] = (raw_eval.reshape(-1).astype(bool) & (~raw_ground.reshape(-1).astype(bool)))

Detailed discussion and visulization can be found as follows:

---

Hanqiu found it when we evaluated and reproduced Floxels in the Argoverse 2 validation set locally.

# for local and online eval mask from official repo
s5cmd --no-sign-request cp "s3://argoverse/tasks/3d_scene_flow/zips/*" .

The provided eval_mask at the first 5-10 frames in each scene will include ground points as evaluation and some flow annotation actually on the ground points.

Some discussions and images:

Maybe that's another reason why local validation sets have lower performance than test set (leaderboard-v2 changed eval_mask and eval frames, so it didn't use the v1 one here). But it should be fair for all methods, since we set all ground points' relative flow=0 for all methods in opensf code. 😅