因?yàn)?d檢測(cè)訓(xùn)練時(shí)間太久，所以想要在mmdet3d上開多機(jī)，發(fā)現(xiàn)加載完標(biāo)注文件pkl/json之后，卡住了，找到如下報(bào)錯(cuò)

其中有個(gè)warning ：using best-guess GPU, 大概率是rank不對(duì)，
找到相關(guān)代碼：

def init_dist(launcher, backend='nccl', **kwargs):if mp.get_start_method(allow_none=True) is None:mp.set_start_method('spawn')if launcher == 'pytorch':_init_dist_pytorch(backend, **kwargs)elif launcher == 'mpi':_init_dist_mpi(backend, **kwargs)elif launcher == 'slurm':_init_dist_slurm(backend, **kwargs)else:raise ValueError(f'Invalid launcher type: {launcher}')def _init_dist_pytorch(backend, **kwargs):# TODO: use local_rank instead of rank % num_gpusrank = int(os.environ['RANK'])local_rank = int(os.environ["LOCAL_RANK"])num_gpus = torch.cuda.device_count()# torch.cuda.set_device(rank % num_gpus)torch.cuda.set_device(local_rank)dist.init_process_group(backend=backend, **kwargs)# device = torch.device("cuda", local_rank)

沒什么問題，按照提示修改torch.cuda.set_device(local_rank)還是不work，
懷疑環(huán)境沒搞對(duì)，增加環(huán)境初始化：

def configure_nccl():import subprocessos.environ["NCCL_LAUNCH_MODE"] = ""os.environ["NCCL_IB_DISABLE"] = "0"os.environ["NCCL_IB_HCA"] = subprocess.getoutput("cd /sys/class/infiniband/ > /dev/null; for i in mlx5_*; ""do cat $i/ports/1/gid_attrs/types/* 2>/dev/null ""| grep v >/dev/null && echo $i ; done; > /dev/null")os.environ["NCCL_IB_GID_INDEX"] = "3"os.environ["NCCL_IB_TC"] = "106"

work!

總結(jié)

以上是生活随笔為你收集整理的mmdetection/mmdetection3d多机多卡训练的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。