108 lines
2.8 KiB
Markdown
108 lines
2.8 KiB
Markdown
|
# BEVFusion项目总结 - 一页纸版本
|
|||
|
|
|
|||
|
|
**日期**: 2025-10-30
|
|||
|
|
**状态**: Phase 4A Stage 1 正在训练 🚀
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📊 项目历程
|
|||
|
|
|
|||
|
|
| 阶段 | Epoch | 配置 | 性能 | 状态 |
|
|||
|
|
|------|-------|------|------|------|
|
|||
|
|
| Phase 1-2 | 1-19 | 基础模型 | Baseline | ✅ |
|
|||
|
|
| Phase 3 | 20-23 | Enhanced头 400×400 | NDS 0.6941, mIoU 0.41 | ✅ |
|
|||
|
|
| Stage 1 | 24-33 | Enhanced头 600×600 | 训练中 | 🔄 |
|
|||
|
|
| Stage 2 | 待定 | Enhanced头 800×800 | 待规划 | ⏸️ |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## ⚠️ 8个关键问题与解决方法
|
|||
|
|
|
|||
|
|
### 1. Docker重启后mmcv无法加载 ⭐⭐⭐
|
|||
|
|
**错误**: `ImportError: libtorch_cuda_cu.so`
|
|||
|
|
**原因**: 库文件命名不匹配
|
|||
|
|
**解决**:
|
|||
|
|
```bash
|
|||
|
|
cd /opt/conda/lib/python3.8/site-packages/torch/lib
|
|||
|
|
ln -sf libtorch_cuda.so libtorch_cuda_cu.so
|
|||
|
|
ln -sf libtorch_cuda.so libtorch_cuda_cpp.so
|
|||
|
|
ln -sf libtorch_cpu.so libtorch_cpu_cpp.so
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 2. 800×800显存不足 ⭐⭐⭐
|
|||
|
|
**错误**: `CUDA out of memory` (18GB/32GB)
|
|||
|
|
**原因**: 分辨率提升4x → 显存需求4x
|
|||
|
|
**解决**: 渐进式训练 (600×600 → 800×800)
|
|||
|
|
|
|||
|
|
### 3. Shape不匹配 ⭐⭐
|
|||
|
|
**错误**: `Target 800×800 vs Input 400×400`
|
|||
|
|
**原因**: output_scope配置错误
|
|||
|
|
**解决**: 修改配置 + 添加自适应插值
|
|||
|
|
|
|||
|
|
### 4. 插值类型错误 ⭐⭐
|
|||
|
|
**错误**: `upsample not implemented for Long`
|
|||
|
|
**原因**: PyTorch不支持整型插值
|
|||
|
|
**解决**: 使用`.float()`插值，保持float
|
|||
|
|
|
|||
|
|
### 5. LD_LIBRARY_PATH未传递 ⭐
|
|||
|
|
**原因**: torchpack可能不传递环境变量
|
|||
|
|
**解决**: 在命令前明确声明环境变量
|
|||
|
|
|
|||
|
|
### 6. DataLoader共享内存 ⭐
|
|||
|
|
**错误**: `unable to write to file </torch_xxx>`
|
|||
|
|
**解决**: `--data.workers_per_gpu 0`
|
|||
|
|
|
|||
|
|
### 7. Python缓存 ⭐
|
|||
|
|
**现象**: 代码修改不生效
|
|||
|
|
**解决**: `find . -name __pycache__ -exec rm -rf {} +`
|
|||
|
|
|
|||
|
|
### 8. 配置未同步 ⭐
|
|||
|
|
**现象**: 复制配置后total_epochs错误
|
|||
|
|
**解决**: 检查所有分辨率和epochs配置
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🎯 当前状态 (13:15)
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
训练: Phase 4A Stage 1
|
|||
|
|
配置: 600×600, 4层Decoder, Deep Sup + Dice
|
|||
|
|
GPU: 4张 @ 100%利用率, ~30GB显存
|
|||
|
|
进度: Epoch 1, iter 100+/30895
|
|||
|
|
Loss: 6.9 → 6.3 (下降中)
|
|||
|
|
ETA: ~9天
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## ✅ Docker重启后3步启动
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# 1. 创建符号链接
|
|||
|
|
cd /opt/conda/lib/python3.8/site-packages/torch/lib && \
|
|||
|
|
ln -sf libtorch_cuda.so libtorch_cuda_cu.so && \
|
|||
|
|
ln -sf libtorch_cuda.so libtorch_cuda_cpp.so && \
|
|||
|
|
ln -sf libtorch_cpu.so libtorch_cpu_cpp.so
|
|||
|
|
|
|||
|
|
# 2. 验证环境
|
|||
|
|
cd /workspace/bevfusion && python -c "from mmcv.ops import nms_match; print('OK')"
|
|||
|
|
|
|||
|
|
# 3. 启动训练
|
|||
|
|
bash START_PHASE4A_STAGE1.sh
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📚 完整文档
|
|||
|
|
|
|||
|
|
**最全面**: `项目进展与问题解决总结_20251030.md` (本文档)
|
|||
|
|
**快速查阅**: `QUICK_REFERENCE_CARD.md`
|
|||
|
|
**索引**: `项目状态总览_20251030.md`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**下一检查点**: Epoch 1验证 (~21小时后)
|
|||
|
|
|
|||
|
|
|
|||
|
|
|