無人駕駛汽車 (Self-Driving Cars)

“If you recognize that self-driving cars are going to prevent car accidents, AI will be responsible for reducing one of the leading causes of death in the world.” — Mark Zuckerberg

“如果您意識到自動駕駛汽車將預防車禍，那么人工智能將負責減少世界上主要的死亡原因之一。” - 馬克·扎克伯格

Whenever we think about the AI world, the auto industry immediately comes to our minds. Self-driving cars is one of the fascinating future which does not seem to be a distant reality. We just sit inside the car and watch a movie while it takes you to its destination.

每當我們想到AI世界時，就會立即想到汽車行業。 自動駕駛汽車是令人著迷的未來之一，這似乎并不是遙不可及的現實。 我們只是坐在車內，一邊看電影，一邊將您帶到目的地。

But is it that easy that cars can drive fully autonomous and pay attention to all contexts in an environment? In the past few years, many papers have been published to predict the possible future trajectories of cars and pedestrians that are socially acceptable.

但是，汽車能夠完全自動駕駛并關注環境中的所有環境是否容易呢？ 在過去的幾年中，已經發表了許多論文來預測社會上可以接受的汽車和行人的未來軌跡。

Question: What will be one of the biggest challenges of self-driving cars?

問題 ：自動駕駛汽車的最大挑戰是什么？

Answer: Understanding pedestrian behavior and their future trajectory.

答案 ：了解行人的行為及其未來軌跡。

Human motion can be described as multimodal i.e. humans have the possibility to move in multiple directions at any given instant of time. And this behavior is one of the biggest challenges of self-driving cars. Since they have to navigate through a human-centric world.

人體運動可以描述為多模式的，即人體可以在任何給定的瞬間沿多個方向移動。 這種行為是自動駕駛汽車的最大挑戰之一。 因為他們必須穿越以人為本的世界 。

In this first part, I will discuss three papers in brief whose main aim is to predict future possible trajectories of pedestrians.

在第一部分中，我將簡要討論三篇論文，其主要目的是預測行人的未來可能軌跡。

社交GAN (Social GAN)

This is one of the initial papers that has stated using GAN to predict the possible trajectories of humans.

這是使用GAN預測人類可能軌跡的最初論文之一。

This paper tries to solve the problem by predicting the socially plausible future trajectories of humans that will help self-driving cars in making the right decision.

本文試圖通過預測人類在社會上可行的未來軌跡來解決該問題，這將有助于自動駕駛汽車做出正確的決定。

目標： (Aim:)

The paper aims at resolving two major challenges:

本文旨在解決兩個主要挑戰：

To have a computationally efficient interaction model among all people in a scene.

在場景中的所有人之間建立計算有效的交互模型。

To learn and produce multiple trajectories that are socially acceptable.

學習并產生社會上可接受的多種軌跡。

方法 (Method)

Fig 1. 圖1. Screenshot from Social Gan Research paperSocial Gan Research論文的屏幕截圖

This paper presented a GAN based encoder-decoder network that consists of a LSTM network for each person and a Pooling module that models interactions among them.

本文提出了一個基于GAN的編解碼器網絡，該網絡由每個人的LSTM網絡和對它們之間的交互進行建模的Pooling模塊組成。

The whole model (shown in Fig 1.) can be represented by 3 different components:

整個模型(如圖1所示)。 可以由3個不同的組件表示：

發電機 (Generator)

The Generator consists of an encoder and a decoder. For each person, the encoder takes input as X_i. It embeds the location of each person and provides it as a fixed-length vector to the LSTM cell at time t.

生成器由編碼器和解碼器組成。 對于每個人，編碼器將輸入作為X_i。 它嵌入每個人的位置，并在時間t將其作為固定長度的向量提供給LSTM單元。

The LSTM weights were shared among all the people in a scene that will help with the pooling module to develop interaction among people.

LSTM權重在場景中的所有人之間共享，這將有助于合并模塊發展人與人之間的互動。

Unlike prior work, this paper has used 2 following approaches:

與先前的工作不同，本文使用了以下兩種方法：

a) For an easy training process during backpropagation, instead of predicting the bivariate Gaussian distribution, the decoder directly produces (x, y) coordinates of the person’s location.

a)為了在反向傳播過程中進行簡單的訓練，解碼器無需預測雙變量高斯分布，而是直接生成人所在位置的(x，y)坐標。

b) Instead of providing the social context directly as the input of the encoder, they have provided it once as input to the decoder. This led to an increase in speed to 16x times.

b)他們沒有將社交環境直接提供為編碼器的輸入，而是將其一次提供為解碼器的輸入。 這導致速度提高到16倍。

鑒別器 (Discriminator)

The discriminator consists of an encoder with LSTM layers for each person. The idea of this discriminator is to distinguish the real trajectories with fake ones.

鑒別器由每個人的帶有LSTM層的編碼器組成。 這種鑒別器的思想是用假的偽造來區分真實的彈道。

Ideally, it should classify the trajectories as “fake” if they are not socially acceptable or possible.

理想情況下，如果軌跡在社會上不可接受或不可能，則應將其歸類為“偽造” 。

合并模塊 (Pooling Module)

Fig2.圖2。 Screenshot from Social GAN Paper社交GAN Paper的屏幕截圖

The basic idea of this approach is shown in Fig2. This method computes the relative position of the person 1 (represented in red) and all other people (represented in blue and green). It is then concatenated with hidden states and processed independently through MLP (multi-layer perception).

這種方法的基本思想如圖2所示。 此方法計算人員1(以紅色表示)與所有其他人(以藍色和綠色表示)的相對位置。 然后將其與隱藏狀態連接起來，并通過MLP(多層感知)進行獨立處理。

Eventually, each element is sequentially pooled to compute a person’s 1 pooling vector P1.

最終，每個元素被順序合并以計算一個人的1個合并向量P1。

This method diminishes the limitation of considering people inside a particular grid (S-Pool Grid in Fig 2.)

這種方法減少了考慮在特定網格(圖2中的S-Pool網格 )內部人員的局限性。

損失 (Losses)

3 different losses are used in this paper:

本文使用了3種不同的損耗：

Adversarial loss: This loss is a typical GAN loss that helps in differentiating real and fake trajectories.

對抗性損失 ：這種損失是典型的GAN損失，有助于區分真實和偽造的軌跡。

L2 Loss: This loss takes the distance between the predicted and ground-truth trajectory and measures how far the generated samples are from real ones.

L2損失 ：該損失占用了預測軌跡與地面真實軌跡之間的距離，并測量了所生成的樣本與真實樣本之間的距離。

Variety Loss: This loss helps in generating multiple different trajectories i.e. multimodal trajectories. The idea is very simple, for each input, N different possible outcomes are predicted by randomly sampling ‘z’ from N(0, 1). Eventually, select the best trajectory that has a minimum L2 value.

品種損失 ：這種損失有助于產生多種不同的軌跡，即多峰軌跡。 這個想法非常簡單，對于每個輸入，通過從N(0，1)中隨機采樣“ z”來預測N個不同的可能結果。 最終，選擇具有最小L2值的最佳軌跡。

Screenshot from Social GAN Paper社交GAN Paper的屏幕截圖

蘇菲：細心的GAN (Sophie: An Attentive GAN)

This paper has extended the work of Social GAN and tried to predict a future path for an agent with the help of both physical and social information.

本文擴展了Social GAN的工作，并嘗試借助物理和社交信息來預測代理的未來之路。

Although the aim is still the same as Social GAN, this paper has added scenic information too with the help of images of each frame.

盡管目標仍然與Social GAN相同，但本文還借助每幀圖像添加了風景信息。

The network learns two types of attention components:

網絡學習兩種類型的注意力組件：

Physical Attention: This attention component helps in paying attention and processing the local and global spatial information of the surrounding. As mentioned in the paper: “For example, when reaching a curved path, we focus more on the curve rather than other constraints in the environment”

身體注意 ：此注意組件有助于關注并處理周圍的局部和全局空間信息。 如論文所述 ： “例如，當到達彎曲路徑時，我們更多地關注曲線而不是環境中的其他約束”

Social Attention: In this component, the idea is to give more attention to the movement and decisions of other agents in the surrounding environment. For example: “when walking in a corridor, we pay more attention to people in front of us rather than the ones behind us”

社會關注 ：在此組件中，其想法是更加關注周圍環境中其他代理的移動和決策。 例如：“ 在走廊上行走時，我們會更加關注前方的人，而不是身后的人 ”

方法 (Method)

Fig3. 圖3。 Screenshot from Sophie Research paper.Sophie Research論文的屏幕截圖。

This paper’s proposed approach has been divided into 3 modules (as shown in Fig3.).

本文提出的方法已被劃分成3個模塊(如圖圖三)。

特征提取器模塊 (Feature Extractor module)

This module extracts features from the input in 2 different forms, first as an image for each frame and second as a state of each agent for each frame at a time ‘t’.

該模塊提取物在2組不同的形式輸入功能， 首先為每個幀的圖像，并且第二 ，因為每個代理在時間t的每個幀的狀態。

To extract visual features from the image, they have used VGGnet-19 as a CNN network. The weights of this network are initialized by ImageNet. And to extract features from the past trajectory of all agents, they use a similar approach as Social GAN and use LSTM as an encoder.

為了從圖像中提取視覺特征，他們將VGGnet-19用作CNN網絡。 該網絡的權重由ImageNet初始化。 為了從所有特工的過去軌跡中提取特征，他們使用與Social GAN類似的方法，并使用LSTM作為編碼器。

To understand the interaction between each agent and capture the influence of each agent trajectory on another agent, the pooling module was used in Social GAN. This paper has mentioned 2 limitations with that method:

為了了解每個代理之間的交互并捕獲每個代理軌跡對另一個代理的影響，在Social GAN中使用了合并模塊。 本文提到了該方法的兩個局限性：

Max function may discard the important features of the inputs as they might lose their uniqueness

最大功能可能會丟棄輸入的重要特征，因為它們可能會失去其唯一性

After the max operation, all the trajectories are concatenated which may lead to having an identical join feature representation.

在max操作之后，所有軌跡被級聯，這可能導致具有相同的聯接特征表示。

Because of these limitations, they define an ordering structure. In this, they use sort as permutation invariant function instead of max(used in Social GAN). They sort the agents by calculating the euclidean distance between the target agent and other agents.

由于這些限制，它們定義了排序結構。 在這種情況下，他們使用sort作為置換不變函數，而不是max (在Social GAN中使用)。 他們通過計算目標代理與其他代理之間的歐式距離來對代理進行排序。

注意模塊 (Attention module)

With the help of physical or social attention, this module helps in highlighting the important information of the input for the next module.

在身體或社會關注的幫助下，該模塊有助于突出顯示下一個模塊輸入的重要信息。

The idea is, as humans pay more attention to certain obstacles or objects in an environment like upcoming turns or people walking towards them, a similar kind of attention needs to be learned.

這個想法是，當人們在即將到來的轉彎或人們朝他們走去的環境中更加關注某些障礙或物體時，需要學習類似的注意力。

As mentioned before, this network tends to learn 2 different attention.

如前所述，該網絡傾向于學習2種不同的注意力。

In physical attention, hidden states of the LSTM from the GAN module and learned features from the visual context is provided as input. This helps in learning more about the physical constraints like the path is straight or curved, what is the current movement direction, position, and more?

在物理注意方面 ，將GAN模塊中LSTM的隱藏狀態和可視上下文中的學習功能作為輸入提供。 這有助于了解更多關于物理約束的信息，例如路徑是直線還是彎曲的，當前的運動方向，位置等等？

In social attention, the LSTM features learned from the feature module together with hidden states of the LSTM from the GAN module are provided as input. This helps in focusing on all agents that are important to predict the further trajectory.

在社會關注中 ，從功能模塊中學習到的LSTM功能以及從GAN模塊中獲得的LSTM隱藏狀態都作為輸入提供。 這有助于專注于對預測進一步軌跡很重要的所有代理。

GAN模塊 (GAN module)

This module takes that highlighted input features to generate a realistic future path for each agent that satisfies all the social and physical norms.

該模塊采用突出顯示的輸入功能，為滿足所有社會和身體規范的每個代理生成現實的未來之路。

This GAN module is majorly inspired by the Social GAN with almost no further changes.

此GAN模塊主要受Social GAN的啟發，幾乎沒有進一步的變化。

The input to the generator is the selected features from the attention module as well as white noise ‘z’ sampled from a multivariate normal distribution.

生成器的輸入是從注意力模塊中選擇的功能以及從多元正態分布中采樣的白噪聲“ z” 。

損失 (Losses)

This approach has used 2 losses which are also similar to Social GAN.

該方法使用了2個損失，這也類似于Social GAN。

Adversarial Loss: This loss helps in learning discrimination between the generated and real samples.

對抗性損失：這種損失有助于了解生成的樣本與真實樣本之間的區別。

L2 Loss: This loss is similar to “variety loss” used in Social GAN.

L2損失：此損失類似于Social GAN中使用的“ 品種損失 ”。

社交方式 (Social Ways)

In this paper, they are also trying to predict the pedestrian’s trajectories and their interaction. However, they are also aiming to solve one problem from all previous approaches: Mode Collapse.

在本文中 ，他們還試圖預測行人的軌跡及其相互作用。 但是，他們還旨在解決以前所有方法中的一個問題： 模式崩潰 。

Mode Collapse is the opposite of multimodality. In this, the generator tries to produce similar samples or the same set of samples leading to similar modes in output.

模式崩潰與多模式相反。 在這種情況下，生成器嘗試生成相似的樣本或同一組樣本，從而導致輸出中的相似模式。

To solve the mode collapse problem, this paper uses info-GAN instead of L2 Loss or variety loss.

為了解決模式崩潰問題，本文使用info-GAN代替了L2損失或品種損失。

方法 (Method)

Fig4.圖4。 Screenshot from Social Ways Research paper社交方式研究論文的屏幕截圖

This method comprises of 3 different components:

此方法包含3個不同的組件：

發電機 (Generator)

The generator consists of an encoder-decoder network. Here the past trajectory of each agent was fed into respective LSTM-E (Encoder), which encodes the history of the agent. The output of each LSTM-E was fed into attention pooling as well as a decoder.

生成器由一個編碼器-解碼器網絡組成。 在這里，每個代理的過去軌跡被輸入到相應的LSTM-E(編碼器)中，該代碼對代理的歷史進行編碼。 每個LSTM-E的輸出都被輸入到注意力集中以及解碼器中。

For decoding, the future trajectories, hidden states from LSTM-E, noise vector ‘z’, latent code ‘c’, and important interacting agent features from the attention pooling are fed into decoder.

對于解碼，未來的軌跡，LSTM-E的隱藏狀態，噪聲矢量“ z”，潛在代碼“ c”以及注意力集中的重要交互代理功能都將輸入解碼器。

The latent code ‘c’ helps in maximizing a lower bound of the mutual information between the distribution of generated output and ‘c’.

潛在代碼“ c”有助于最大化生成的輸出與“ c”的分布之間的相互信息的下限。

注意池 (Attention Pooling)

This paper uses a similar kind of approach as used in Sophie: an attentive GAN.

本文使用的方法與Sophie中使用的方法類似：殷勤的GAN 。

However, in addition to the euclidean distance between agents (used in Sohpie), 2 more features are being used:

但是，除了代理之間的歐式距離(用于Sohpie)之外，還使用了2個功能：

Bearing angle: “the angle between the velocity vector of agent 1 and vectors joining agents 1 and agent 2.”

方位角 ： “代理1的速度矢量與連接代理1和代理2的矢量之間的角度。”

The distance of closest approach: “the smallest distance, 2 agents would reach in the future if both maintain their current velocity.”

最接近的距離 ： “最小的距離，如果兩個代理都保持當前速度，將來會到達。”

Instead of sorting, the attention weights were obtained by scalar product and softmax operation between hidden states and the above-mentioned three features.

代替排序，注意力權重通過隱藏狀態與上述三個特征之間的標量積和softmax操作獲得。

鑒別器 (Discriminator)

Discriminator consists of LSTM based encoder with multiple dense layers. Generated trajectories from the generator for each agent and ground-truth trajectories were fed into the discriminator.

鑒別器由具有多個密集層的基于LSTM的編碼器組成。 生成器為每個代理生成的軌跡和真實的軌跡被輸入到鑒別器中。

As output, the probability that generated trajectories are real is provided.

作為輸出，提供了生成的軌跡是真實的概率。

損失 (Losses)

There are 2 losses used in this process:

此過程中使用了2種損失：

Adversarial Loss: This is the normal GAN loss that helps in differentiating between real and fake samples.

對抗性損失 ：這是正常的GAN損失，有助于區分真實和偽造樣品。

Information Loss: The basic idea of this loss is to maximize mutual information. And that is achieved by minimizing the negative- loss likelihood on salient variable ‘c’.

信息丟失 ：這種丟失的基本思想是最大化相互信息。 這是通過使顯著變量'c'的負損失可能性最小化來實現的 。

結果 (Results)

All three papers have tried to learn from previous approaches and have gained some new insights.

所有這三篇論文都試圖向以前的方法學習，并獲得了一些新見解。

There are 2 metrics that are used to evaluate this application:

有兩個用于評估此應用程序的指標：

Average Displacement Error (ADE): It is the average L2 distance on all predicted timesteps between the generated trajectory and ground-truth trajectory.

平均位移誤差(ADE) ：它是在生成的軌跡和地面真實軌跡之間的所有預測時間步長上的平均L2距離。

Final Displacement Error(FDE): This is the smallest distance between the generated trajectory and ground-truth trajectory at the final predicted timestep.

最終位移誤差(FDE) ：這是在最終預測時間步中生成的軌跡與地面真實軌跡之間的最小距離。

There are 5 datasets, which are used as benchmarking this application. All these approaches have been tested on all these datasets and they have provided valuable comparable results.

有5個數據集，用作基準測試此應用程序。 所有這些方法都在所有這些數據集上進行了測試，它們提供了有價值的可比結果。

Fig5.圖5。 Screenshot of Results from Social Ways社交方式結果的屏幕截圖

From Fig5. , it can be said that all three approaches show promising results on some datasets. However, I think with hyperparameter tuning and little adjustments, results may shuffle too. I believe all three have certain advantages and could be used to further research in this area.

從圖5。 ，可以說這三種方法在某些數據集上都顯示出令人鼓舞的結果。 但是，我認為通過超參數調整和少量調整，結果也可能會混亂。 我相信這三者都具有一定的優勢，可以用于該領域的進一步研究。

結論 (Conclusion)

The problem of modeling human motion prediction in a scene along with human-human interaction is challenging yet vital for self-driving cars. Without modeling this behavior, it is impossible that self-driving cars could be fully operational.

在場景中對人的運動預測進行建模以及人與人之間的交互作用的問題是具有挑戰性的，但對于自動駕駛汽車而言卻至關重要。 如果不對此行為進行建模，那么自動駕駛汽車就不可能完全運轉。

How to model human-human interaction is the major difference between the above-mentioned approaches. From the theory and suggested methods, I believe attention on distance and the bearing angle between 2 agents is one the most crucial ways to move forward.

如何模擬人與人之間的互動是上述方法之間的主要區別。 從理論和建議的方法來看，我認為關注距離和兩個代理之間的方位角是前進的最關鍵方法之一。

But it is completely my perspective. There could be multiple ways that this could be implemented and enhanced. And we will see that in Part 2 too.

但這完全是我的觀點。 可能有多種方法可以實現和增強它。 我們還將在第2部分中看到這一點。

With self-driving cars as?the?focus, I would continue with more approaches in Part 2 with a focus on the trajectory prediction of cars.

以自動駕駛汽車為重點，我將在第2部分中繼續介紹更多方法，重點是汽車的軌跡預測。

I am happy for any further discussion on this paper and in this area. You can leave a comment here or reach out to me on my LinkedIn profile.

對于本文和該領域的任何進一步討論，我感到很高興。 您可以在此處發表評論，也可以通過我的LinkedIn個人資料與我聯系。

翻譯自: https://towardsdatascience.com/trajectory-prediction-self-driving-cars-ai-40a7c6eecb4c

總結

以上是生活随笔為你收集整理的轨迹预测演变（第1/2部分）的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。