132 lines
3.0 KiB
Markdown
132 lines
3.0 KiB
Markdown
# 环境问题记录 - Phase 4A启动失败
|
||
|
||
**时间**: 2025-10-30
|
||
**问题**: 无法启动Phase 4A训练
|
||
**错误**: ImportError: libtorch_cuda_cu.so: cannot open shared object file
|
||
|
||
---
|
||
|
||
## 错误详情
|
||
|
||
### 完整错误信息
|
||
```
|
||
ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
|
||
|
||
File "/opt/conda/lib/python3.8/site-packages/mmcv/ops/assign_score_withk.py", line 5, in <module>
|
||
ext_module = ext_loader.load_ext(
|
||
```
|
||
|
||
### 已尝试的解决方案
|
||
|
||
1. ❌ 设置LD_LIBRARY_PATH
|
||
```bash
|
||
export LD_LIBRARY_PATH=/opt/conda/lib:/opt/conda/lib/python3.8/site-packages/torch/lib:$LD_LIBRARY_PATH
|
||
```
|
||
|
||
2. ❌ 使用完整路径
|
||
```bash
|
||
/opt/conda/bin/torchpack dist-run -np 6 /opt/conda/bin/python tools/train.py
|
||
```
|
||
|
||
3. ❌ 使用torch.distributed.launch
|
||
```bash
|
||
python -m torch.distributed.launch --nproc_per_node=6
|
||
```
|
||
|
||
---
|
||
|
||
## 对比: Phase 3成功环境
|
||
|
||
**Phase 3训练 (成功)**:
|
||
- 时间: 2025-10-21 ~ 2025-10-29
|
||
- 命令: 与Phase 4A相同格式
|
||
- 结果: 稳定运行8天，23 epochs完成
|
||
- 环境: 相同的Docker容器
|
||
|
||
**疑问**:
|
||
- Phase 3和Phase 4A之间环境发生了什么变化？
|
||
- 是否系统更新或Docker重启？
|
||
|
||
---
|
||
|
||
## 诊断建议
|
||
|
||
### 立即检查
|
||
```bash
|
||
# 1. 检查Docker容器状态
|
||
docker ps
|
||
|
||
# 2. 检查是否需要重启Docker
|
||
# (如果之前重启过系统或Docker)
|
||
|
||
# 3. 检查Python环境
|
||
which python
|
||
python --version
|
||
python -c "import torch; print(torch.__version__)"
|
||
|
||
# 4. 检查库文件
|
||
find /opt/conda -name "libtorch_cuda_cu.so"
|
||
ls -l /opt/conda/pkgs/pytorch-*/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
|
||
|
||
# 5. 测试简单导入
|
||
python -c "from mmcv.ops import nms_match"
|
||
```
|
||
|
||
### 可能的解决方案
|
||
|
||
**方案1: 重启Docker容器**
|
||
```bash
|
||
# 退出容器
|
||
exit
|
||
|
||
# 重新进入
|
||
docker start [容器ID]
|
||
docker exec -it [容器ID] /bin/bash
|
||
cd /workspace/bevfusion
|
||
|
||
# 重试
|
||
bash start_phase4a_bev2x_fixed.sh
|
||
```
|
||
|
||
**方案2: 重新编译mmcv (最后手段)**
|
||
```bash
|
||
# 这会比较耗时 (1-2小时)
|
||
pip uninstall mmcv-full -y
|
||
pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html
|
||
```
|
||
|
||
**方案3: 使用不同的启动方式**
|
||
```bash
|
||
# 如果torchpack有问题，直接用PyTorch DDP
|
||
python -m torch.distributed.run --nproc_per_node=6 --master_port=29500 \
|
||
tools/train.py configs/.../multitask_BEV2X_phase4a.yaml \
|
||
--launcher pytorch \
|
||
--load_from runs/enhanced_from_epoch19/epoch_23.pth
|
||
```
|
||
|
||
---
|
||
|
||
## 临时措施
|
||
|
||
### 如果无法立即解决
|
||
|
||
可以先执行其他工作:
|
||
1. 分析Phase 3的详细结果
|
||
2. 准备实车数据采集计划
|
||
3. 研究MapTR集成方案
|
||
4. 设计模型压缩方案
|
||
|
||
### 或降级方案
|
||
|
||
如果BEV 2x实在无法启动，可以考虑:
|
||
1. BEV 1.5x (0.2m分辨率) - 降低50%计算量
|
||
2. 继续Phase 3训练更多epochs
|
||
3. 直接进入Phase 4B模型压缩
|
||
|
||
---
|
||
|
||
**状态**: 待解决
|
||
**优先级**: P0 (最高)
|
||
**影响**: 阻塞Phase 4A训练
|
||
**建议**: 优先解决环境问题，这是后续工作的基础
|