[리뷰] Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?

이번에는 NeurIPS 2022에 발표된 논문인 Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?를 읽고, 리뷰해보고자 합니다.

Index
1. Background
    1.1. Homography
    1.2. Graph Neural Network
    1.3. Bi-Partite Matching
2. Abstract
3. Introduction
4. Related Work
5. Method
    5.1. Overview
    5.2. Data-driven Homography Estimation
    5.3. Forecasting
    5.4. Tracking via Forecasting
6. Experiment
7. Conclusion 

1. Background

1.1. Homography

Homography

1.2. ByteTrack

Graph Neural Network

1.3. ByteTrack

Bi-Partite Matching

 

2. Abstract

• 최근 MOT tracker의 많은 발전이 있었지만, 여전히 3초보다 긴 occlusion의 10% 미만에 대해서만 association를 정상 수행
  • occlusion이 길어지면 association에 대한 검색 공간이 커짐
• missing key가 미래의 trajectory에 대한 근거라고 제안
• moving agent에 대한 작지만 다양한 trajectory 예측 세트가 association에 대한 검색 공간을 크게 감소시켜, long-term track에 대해 robust 해짐

 

3. Introduction

• 최근 SOTA를 달성한 tracker는 regression과 combinatorial optimization을 re-id model과 결합
  • 짧은 occlusion에는 효과적
  • occlusion이 길어질수록 가능한 association 집합이 증가하며, 이에따라 3초 이상 지속되는 occlusion에 대해 10% 정도만 제대로 tracking함
    • 따라서 re-id model은 occlusion을 해결하기에 적합하지 않음
• 인간의 움직임은 관찰할 수 없는 요소에 의해 이루어지기 때문에, 단순한 선형모델로 추측하기 힘듬
• association 집합 search space를 줄이기 위해, 장기 trajectory 예측 제안
  • tracking시 perspective projection의 영향을 받지 않지만, bird’s-eye view에서 효과를 발휘
• bird’s-eye view에서 2D bounding box의 localization은 시간적으로 일관성 필요
  • data-driven 방식으로 sequence당 하나의 homography를 추정하여 해결

 

4. Related Work

• multi-object tracking
  • 초기의 tracker는 association의 최적화와 hand-crafting 및 motion 기반의 descriptor에 초점
  • 최근의 tracker는 data 중심이며, 주로 appearance에 초점
  • regression을 이용한 방법은 detection이 누락되었을 때도 object에 대해 localization이 가능하며, 짧은 시간의 occlusion을 해결하기 위해 re-id model과 함께 사용
    • Tracking without bells and whistles, How to train your deep multi-object tracker, Tracking objects as points, …, etc 
    • 그러나 occlusion 이후, object의 속도가 빠른 경우 tracking 실패
  • 긴 시간의 occlusion을 해결하기 위해 graph neural network를 이용한 방법이 있지만, 계산 cost가 높음
    • Learning a neural solver for multiple object tracking, GNN3DMOT, Learnable Onine Graph Representations for 3D Multi-Object Tracking, …, etc
• trajectory forecasting
  • 초기의 forecasting은 physical model을 기반으로 한 결정론적 방법
  • 후기의 forecasting은 주위 상황과 상호작용을 모델링하는데 중점을 둔 LSTM기반의 encoder-decoder를 사용한 방법
  • 가장 최근의 forecasting은 가능한 미래 trajectory의 분포를 배우는 GAN을 사용한 방법
    • Social GAN, Social-BiGAT, Social Ways, MG-GAN, Goal-GAN, …, etc
  • 최근 non-linear modeling을 위해 보행자의 목표, 의도에 대한 예측과 다양한 trajectory를 산출하기 위한 multimodal posterior distribution을 추정하는데 노력

 

5. Method

5.1. Overview

• sequence 전체에 대한 이미지 평면을 3D로 매핑하는 homography 행렬 $H$ 추정
• tracker의 output과 $H$를 이용하여 bird’s-eye view에서의 tracking을 수행하도록 model 통합
• trajectory forecasting network를 사용하여 bird’s-eye view에서의 손실된 track 예측
• 예측을 online tracker에 통합하고, 새로운 탐지를 기존 track에 연결해 occlusion 해결

5.2. Data-driven Homography Estimation

• monocular object tracking과 forecasting을 결합하기 위해 sequence에서 detection과 track을 bird’s-eye view space의 좌표로 변환이 필요함
• 이미지 평면에서 2D object detection을 고려했을 때, 하단 중심의 위치를 2D bird’s-eye view space의 좌표에 매핑하는 homography $H$를 찾음
  • sequence의 첫 번째 frame의 추정되거나 알려진 3D point cloud를 재구축 하기 위해 depth estimator를 synthetic dataset을 이용하여 학습
  • semantic segmentation network를 이용하여 평면 영역을 mask, select하여 지면에 맞춤
  • 지면의 법선 vector를 추정하고, 평면을 XY평면에 정렬
  • Z축을 따라 접지된 지점을 projection하여 지면의 지점과 bird’s-eye view space의 좌표사이에 대응 관계를 만들어 $H$ 추정
  • 지평선 근처의 픽셀에 대한 homography 변환을 선형화하여 변환의 발산 방지

• Static camera
  • 첫 번째 frame에 대해서만 $H$ 계산하여 이용
• Moving camera
  • 각 frame에 대해 $H_{t}$를 계산하여 이용
  • optical flow를 사용하여 지면 픽셀간의 대응 관계를 찾아내고, 이를 이용하여 두 점 집합의 translation vector를 계산
  • 경험적으로, rotation없이 translation을 추정하는 것이 더 robust한 camera의 motion을 계산한다는 것을 발견

5.3. Forecasting

• bird’s-eye view에서 object의 localization은 미래 trajectory를 forecasting할 수 있도록 함
  • 고정된 크기의 과거 trajectory의 GT가 필요
  • forecasting된 trajectory는 noisy하고 길이가 다양함
• forecasting model은 많은 prediction을 장려하고, L2 distance를 최소화하는 metric에 최적화
• Preprocessing
  • 고정된 크기의 관측된 trajectory를 입력으로 사용하는 LSTM encoder-decoder 구조 사용
    • bird’s-eye view에서 localization된 동일한 track ID의 detection을 이용하여 입력 trajectory 구성
  • localization noise를 보정하기 위해 kalman filter를 사용하여 noisy한 관측을 smoothing 하고, trajectory를 linearly extrapolation 하여 필요한 고정된 입력 길이를 얻음
• Trajectory forecasting design patterns
  • 실험 에서 아래의 주요 설계 패턴을 포함
  • Stochasticity
    • 확률적 trajectory predictor를 사용하여 미래 trajectory의 불확실성을 고려하기 위해 가능한 여러 trajectory를 sampling 할 수 있음
    • Social GAN의 접근 방식을 따르고 GAN 모델을 학습하고 다양한 많은 sample들의 loss로 학습
  • Social Interactions
    • 주위 상황은 object의 미래 motion에 영향을 미침
    • pooling, attention, graph neural network를 활용하여 상호작용 예측
    • tracking에서 상호작용이 중요한지 파악하기 위해 Social GAN활용
  • Multimodality and Diversity
    • multi-generator GAN에서 가능한 적은 sample을 이용하여 여러 decoder head를 학습 후, 하나의 장면에서 다양한 예측을 수행

5.4. Tracking via Forecasting

• visible objects를 tracking 할 수 있는 tracker가 있다고 가정
  • 새로운 detection에 의해 track이 update되면 활성화 된 것으로 간주하고 활성화 세트 $S_{A}$에 유지, 그렇지 않으면 비활성화 세트 $S_{I}$에 유지
  • 각 frame $t$에 대해 tracker는 track $\mathcal{O}=(o_{1}, \dots , o_{M})$의 output을 가짐
    • $o_{i}=(\text{ID}_{i}, b_{i}, f_{i})$
      • $ID \in \mathbb{N}^{+}$, $b \in \mathbb{R}^{4}$, $f \in \mathbb{R}^{D}$
      • ID는 track의 id, b는 pixel space의 bounding box, f는 D차원의 appearance feature vector
• 추정된 $H$를 사용하여 bird’s-eye view space($x \in \mathbb{R}^{2}$)의 bounding box를 localization
• Quo Vadis?
  • track이 일시적으로 손실되어 비활성화 되면 해당 track을 memory bank로 이동 후, trajectory 예측 model을 사용하여 $k$개의 길이가 $\tau_{max}$인 trajectory 예측
    • 다시 활성화 될 때 까지 예측된 trajectory를 따라 이동하며, 각 frame 마다 완전히 새로운 trajectory를 예측하지 않음
• Filtering and removing predictions
  • 확률적으로 여러 trajectory의 예측을 하는 경우, 부정확하거나 오래된 예측을 제한하여 잘못된 association의 가능성을 줄임
    • 예측 trajectory의 최대 수명을 $\tau_{max}$로 제한
    • 비활성 track의 모든 예측이 $\tau_{max}$ 이전에 제거되면, 전체 track을 제거
    • 결국에는 visible object가 detection된다고 가정하고, $\tau_{vis}$ frame 이상에 대한 예측을 제거
  • 카메라에 더 가까운 object와의 overlap이 25% 이하인 경우, visible object로 간주
• Matching predictions with new detections
  • 비활성 track $i$와 새로운 detection $j$는 기존의 방법에 따라 bi-partitie matching을 이용하여 match하며, 이때의 association cost $c_{ij}$는 아래와 같음
    • $c_{ij}=(\Delta_{IoU}+\max(\tau_{L_{2}}-\Delta_{L_{2}}, 0))\cdot(\Delta_{App} \geq \tau_{App} \text{ and } \Delta_{IoU} \geq \tau_{IoU})$
      • $\Delta_{IoU}$는 두 bounding box 사이의 IoU score, $\Delta_{L_{2}}$는 bird’s-eye view에서 prediction $i$와 detection $j$간의 euclidean distance, $\Delta_{App}$은 visual feature $f_{i}$와 $f_{j}$의 cosine distance, $\tau_{L_{2}}$, $\tau_{IoU}$, $\tau_{App}$은 matching을 위한 threshold
    • 잘못된 association을 줄이기 위해 시각적으로 일치하지 않는 association에 대한 필터 역할을 하는 최소한의 $\tau_{App}$과 $\tau_{IoU}$가 필요
  • 실제 공간의 spatial distance를 기반으로 association하면 recall이 높아지고 IDSW가 줄어들지만, 복잡한 장면에서 ID transfer error가 증가할 수 있음

 

6. Experiment

• MOT17 validation dataset에서의 사용한 요소에 따른 성능 차이

• threshold 사용 유무에 따른 성능 차이

• 다양한 tracker에 제시된 알고리즘을 사용하였을 때, 성능 향상

• MOT17, MOT20의 test dataset에서의 성능 비교

 

7. Conclusion 

• 조합된 model을 이용하여 구축한 bird’s-eye view에서 여러 가지의 실현 가능한 trajectory 예측을 활용하는 이점을 보여줌
  • 특히 긴 시간 동안의 occlusion 이후, object ReID에 중점
• trajectory 예측 model을 MOT와 결합
• 아래와 같은 한계 존재
  • monocular video와 trajectory 예측을 바탕으로 bird’s-eye view에서 trajectory를 구축해야하므로, 모델이 복잡함
  • bird’s-eye view를 구축하기 위해 homography 변환을 이용하는데, homography 행렬이 추정의 품질에 의존하므로 3D localization 오류에 민감

 

 

논문 링크

https://arxiv.org/abs/2210.07681

https://github.com/dendorferpatrick/QuoVadis