124 lines
2.3 KiB
Markdown
124 lines
2.3 KiB
Markdown
|
# Phase 4A Stage 1 - 当前8卡训练配置
|
|||
|
|
|
|||
|
|
**更新**: 2025-11-01 12:20
|
|||
|
|
**状态**: ✅ 训练进行中
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🚀 关键配置
|
|||
|
|
|
|||
|
|
### 硬件
|
|||
|
|
```
|
|||
|
|
GPU: 8×Tesla V100S-32GB
|
|||
|
|
显存: 28.8-29.3GB/GPU (88-89%利用率)
|
|||
|
|
温度: 44-47°C
|
|||
|
|
功耗: 65-70W/GPU
|
|||
|
|
状态: 100%满载
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 训练参数
|
|||
|
|
```
|
|||
|
|
配置文件: configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/multitask_BEV2X_phase4a_stage1.yaml
|
|||
|
|
启动脚本: START_FROM_EPOCH1.sh
|
|||
|
|
分布式: torchpack dist-run -np 8
|
|||
|
|
Batch Size: 1/GPU (总batch=8)
|
|||
|
|
分辨率: 600×600 GT
|
|||
|
|
Epochs: 10
|
|||
|
|
输出目录: /data/runs/phase4a_stage1/
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 性能指标
|
|||
|
|
```
|
|||
|
|
训练速度: 2.67秒/迭代
|
|||
|
|
单epoch: ~11小时
|
|||
|
|
10 epochs: ~9.5天
|
|||
|
|
加速比: 1.7× (vs 4卡18天)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📂 重要路径
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# 配置文件
|
|||
|
|
/workspace/bevfusion/configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/multitask_BEV2X_phase4a_stage1.yaml
|
|||
|
|
|
|||
|
|
# 启动脚本
|
|||
|
|
/workspace/bevfusion/START_FROM_EPOCH1.sh
|
|||
|
|
|
|||
|
|
# 输出目录
|
|||
|
|
/data/runs/phase4a_stage1/
|
|||
|
|
|
|||
|
|
# 预训练模型
|
|||
|
|
/data/pretrained/swint-nuimages-pretrained.pth
|
|||
|
|
|
|||
|
|
# 初始权重
|
|||
|
|
/data/runs/phase4a_stage1/epoch_1.pth
|
|||
|
|
|
|||
|
|
# 训练日志
|
|||
|
|
/workspace/bevfusion/phase4a_stage1_new_*.log
|
|||
|
|
|
|||
|
|
# 文档目录
|
|||
|
|
/workspace/bevfusion/project/
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🔍 监控命令
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
# GPU状态
|
|||
|
|
nvidia-smi
|
|||
|
|
|
|||
|
|
# 训练进度
|
|||
|
|
tail -100 $(ls -t /workspace/bevfusion/phase4a_stage1_new_*.log | head -1) | grep "Epoch \[1\]\[" | tail -5
|
|||
|
|
|
|||
|
|
# 磁盘空间
|
|||
|
|
df -h | grep -E "/workspace|/data"
|
|||
|
|
|
|||
|
|
# 进程状态
|
|||
|
|
ps aux | grep "train.py" | grep -v grep | wc -l
|
|||
|
|
|
|||
|
|
# Checkpoints
|
|||
|
|
ls -lh /data/runs/phase4a_stage1/*.pth
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📊 当前进度
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Epoch: 1/10
|
|||
|
|
迭代: 4000/15448 (25.9%)
|
|||
|
|
Loss: 2.63-2.78 (下降中)
|
|||
|
|
学习率: 2.000e-05
|
|||
|
|
IOU: 0.618-0.623
|
|||
|
|
|
|||
|
|
预计完成:
|
|||
|
|
- Epoch 1: 2025-11-02 20:00
|
|||
|
|
- Epoch 10: 2025-11-10 20:00
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 📖 详细文档
|
|||
|
|
|
|||
|
|
- **完整配置**: `project/docs/Phase4A_Stage1_8GPU配置_20251101.md`
|
|||
|
|
- **快速参考**: `project/docs/8卡训练快速参考.md`
|
|||
|
|
- **项目总览**: `project/README.md`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 🎯 优化要点
|
|||
|
|
|
|||
|
|
1. ✅ **8卡加速**: 从4卡18天→8卡9.5天
|
|||
|
|
2. ✅ **磁盘管理**: evaluation.interval 1→5
|
|||
|
|
3. ✅ **输出路径**: work_dir → /data分区
|
|||
|
|
4. ✅ **显存优化**: batch=1, workers=0
|
|||
|
|
5. ✅ **稳定运行**: 无OOM，无异常
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**训练将在2025-11-10完成！** 🚀
|
|||
|
|
|