'grad_norm': 0.0

[ctcnb]

报错信息

/home/jizhi/anaconda3/envs/llm/lib/python3.10/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
{'loss': 24435.127, 'grad_norm': 0.0, 'learning_rate': 0.0, 'epoch': 0.05}
{'loss': 3054.3909, 'grad_norm': 0.0, 'learning_rate': 3.0303030303030305e-07, 'epoch': 0.09}
{'loss': 0.0, 'grad_norm': 0.0, 'learning_rate': 6.060606060606061e-07, 'epoch': 0.14}
{'loss': 9163.1729, 'grad_norm': 0.0, 'learning_rate': 9.090909090909091e-07, 'epoch': 0.18}

脚本

export PYTHONPATH=$PYTHONPATH:/home/shiym/projects/MedM-VL

deepspeed --include localhost:0 --master_port 29501 lvlm/train.py \
    --deepspeed scripts/train/utils/zero2.json \
    --data_path /home/jizhi/code/ad/data/data.json \
    --conv_version qwen2 \
    --image_path /home/jizhi/code/data/ad \
    --training_recipe common \
    --tune_type_llm lora \
    --llm_lora_r 128 \
    --llm_lora_alpha 256 \
    --llm_lora_dropout 0.05 \
    --llm_lora_bias none \
    --tune_type_encoder_image frozen \
    --tune_type_connector_image full \
    --bf16 True \
    --gradient_checkpointing True \
    --output_dir work_dirs/MedM-VL-2D-3B-en-finetune \
    --resume_from_checkpoint /home/jizhi/code/models/shiym2000/MedM-VL-2D-3B-en \
    --dataloader_num_workers 2 \
    --dataloader_pin_memory True \
    --dataloader_persistent_workers True \
    --num_train_epochs 100 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --learning_rate 2e-5 \
    --weight_decay 0.0 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type cosine \
    --eval_strategy no \
    --save_strategy no \
    --report_to tensorboard \
    --logging_steps 1

[shiym2000]

Please provide the JSON file corresponding to data_path so that we can obtain more information.

[ctcnb]

{
        "id": "xxx",
        "image": [
            "/home/jizhi/code/data/ad/xxx/000100.jpg",
            "/home/jizhi/code/data/ad/xxx/000200.jpg",
            "/home/jizhi/code/data/ad/xxx/000300.jpg",
        ],
        "conversations": [
            {
                "from": "human",
                "value": "请诊断这位患者患有什么疾病？\n<image>\n<image>\n<image>\n"
            },
            {
                "from": "gpt",
                "value": "MCI"
            }
        ]
    },

[shiym2000]

Try setting the learning_rate to 2e-4 and observe if the training effect improves.

[ctcnb]

尝试将学习率设置为 2e-4 并观察训练效果是否提升。

/home/jizhi/anaconda3/envs/llm/lib/python3.10/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
这句话是否有影响,我对训练参数进行了打印

[2025-10-12 21:34:38,096] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-10-12 21:34:39,951] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
[2025-10-12 21:34:41,118] [WARNING] [runner.py:220:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2025-10-12 21:34:41,118] [INFO] [runner.py:610:main] cmd = /home/jizhi/anaconda3/envs/llm/bin/python3.10 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29501 --enable_each_rank_log=None lvlm/train.py --deepspeed scripts/train/utils/zero2.json --data_path /home/jizhi/code/ad/data/ad_face_train_qwen.json --conv_version qwen2 --image_path / --training_recipe common --tune_type_llm lora --llm_lora_r 128 --llm_lora_alpha 256 --llm_lora_dropout 0.05 --llm_lora_bias none --tune_type_encoder_image frozen --tune_type_connector_image full --bf16 True --gradient_checkpointing True --output_dir work_dirs/MedM-VL-2D-3B-en-finetune --resume_from_checkpoint /home/jizhi/code/models/shiym2000/MedM-VL-2D-3B-en --dataloader_num_workers 2 --dataloader_pin_memory True --dataloader_persistent_workers True --num_train_epochs 100 --per_device_train_batch_size 2 --gradient_accumulation_steps 8 --learning_rate 2e-5 --weight_decay 0.0 --warmup_ratio 0.03 --lr_scheduler_type cosine --eval_strategy no --save_strategy no --report_to tensorboard --logging_steps 1
[2025-10-12 21:34:42,270] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-10-12 21:34:44,148] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
[2025-10-12 21:34:45,330] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0]}
[2025-10-12 21:34:45,330] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=1, node_rank=0
[2025-10-12 21:34:45,330] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2025-10-12 21:34:45,330] [INFO] [launch.py:164:main] dist_world_size=1
[2025-10-12 21:34:45,330] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0
[2025-10-12 21:34:45,330] [INFO] [launch.py:256:main] process 812850 spawned with command: ['/home/jizhi/anaconda3/envs/llm/bin/python3.10', '-u', 'lvlm/train.py', '--local_rank=0', '--deepspeed', 'scripts/train/utils/zero2.json', '--data_path', '/home/jizhi/code/ad/data/ad_face_train_qwen.json', '--conv_version', 'qwen2', '--image_path', '/', '--training_recipe', 'common', '--tune_type_llm', 'lora', '--llm_lora_r', '128', '--llm_lora_alpha', '256', '--llm_lora_dropout', '0.05', '--llm_lora_bias', 'none', '--tune_type_encoder_image', 'frozen', '--tune_type_connector_image', 'full', '--bf16', 'True', '--gradient_checkpointing', 'True', '--output_dir', 'work_dirs/MedM-VL-2D-3B-en-finetune', '--resume_from_checkpoint', '/home/jizhi/code/models/shiym2000/MedM-VL-2D-3B-en', '--dataloader_num_workers', '2', '--dataloader_pin_memory', 'True', '--dataloader_persistent_workers', 'True', '--num_train_epochs', '100', '--per_device_train_batch_size', '2', '--gradient_accumulation_steps', '8', '--learning_rate', '2e-5', '--weight_decay', '0.0', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--eval_strategy', 'no', '--save_strategy', 'no', '--report_to', 'tensorboard', '--logging_steps', '1']
******************************Stage 1******************************
Load args...
[2025-10-12 21:34:49,427] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-10-12 21:34:50,411] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
[2025-10-12 21:34:50,420] [INFO] [comm.py:675:init_distributed] cdb=None
[2025-10-12 21:34:50,420] [INFO] [comm.py:706:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Random seed set to 42 for all libraries.
******************************Stage 2******************************
Load config and model...
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.32it/s]
Some weights of Qwen2ForCausalLM were not initialized from the model checkpoint at /home/jizhi/code/models/Qwen/Qwen2.5-3B-Instruct and are newly initialized: ['lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.60it/s]
Some weights of LVLMForConditionalGeneration were not initialized from the model checkpoint at /home/jizhi/code/models/shiym2000/MedM-VL-2D-3B-en and are newly initialized: ['llm.lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
******************************Stage 3******************************
Save args and config...
******************************Stage 4******************************
Load training_recipe...
******************************Stage 5******************************
Create data_module...
******************************Stage 6******************************
Create trainer and train...

==================================================
可训练参数统计
==================================================

LLM参数:
  base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.2.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.2.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.2.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.2.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.3.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.3.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.3.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.4.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.4.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.4.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.4.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.5.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.5.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.5.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.5.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.6.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.6.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.6.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.6.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.7.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.7.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.7.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.7.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.8.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.8.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.8.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.8.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.9.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.9.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.9.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.9.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.10.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.10.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.10.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.10.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.11.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.11.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.11.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.11.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.12.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.12.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.12.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.12.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.13.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.13.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.13.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.13.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.14.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.14.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.14.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.14.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.15.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.15.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.15.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.15.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.16.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.16.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.16.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.16.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.17.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.17.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.17.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.17.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.18.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.18.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.18.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.18.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.19.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.19.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.19.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.19.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.20.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.20.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.20.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.20.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.21.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.21.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.21.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.21.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.22.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.22.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.22.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.22.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.23.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.23.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.23.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.23.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.24.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.24.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.24.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.24.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.25.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.25.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.25.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.25.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.26.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.26.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.26.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.26.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.27.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.27.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.27.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.27.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.28.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.28.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.28.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.28.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.29.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.29.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.29.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.29.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.30.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.30.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.30.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.30.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.31.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.31.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.31.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.32.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.32.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.32.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.32.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.33.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.33.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.33.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.33.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.34.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.34.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.34.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.34.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768
  base_model.model.model.layers.35.self_attn.q_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.35.self_attn.q_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([2048, 128]), numel=262,144
  base_model.model.model.layers.35.self_attn.v_proj.lora_A.default.weight: requires_grad=True, shape=torch.Size([128, 2048]), numel=262,144
  base_model.model.model.layers.35.self_attn.v_proj.lora_B.default.weight: requires_grad=True, shape=torch.Size([256, 128]), numel=32,768

连接器参数:
  connector.0.weight: requires_grad=True, shape=torch.Size([2048, 768]), numel=1,572,864
  connector.0.bias: requires_grad=True, shape=torch.Size([2048]), numel=2,048
  connector.2.weight: requires_grad=True, shape=torch.Size([2048, 2048]), numel=4,194,304
  connector.2.bias: requires_grad=True, shape=torch.Size([2048]), numel=2,048

图像编码器参数:
  encoder.vision_model.embeddings.patch_embedding.weight: requires_grad=False, shape=torch.Size([768, 3, 16, 16]), numel=589,824
  encoder.vision_model.embeddings.patch_embedding.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.embeddings.position_embedding.weight: requires_grad=False, shape=torch.Size([256, 768]), numel=196,608
  encoder.vision_model.encoder.layers.0.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.0.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.0.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.0.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.0.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.0.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.0.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.0.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.0.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.1.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.1.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.1.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.1.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.1.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.1.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.1.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.1.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.2.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.2.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.2.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.2.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.2.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.2.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.2.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.2.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.3.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.3.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.3.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.3.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.3.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.3.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.3.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.3.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.4.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.4.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.4.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.4.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.4.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.4.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.4.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.4.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.5.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.5.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.5.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.5.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.5.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.5.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.5.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.5.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.6.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.6.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.6.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.6.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.6.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.6.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.6.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.6.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.7.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.7.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.7.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.7.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.7.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.7.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.7.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.7.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.8.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.8.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.8.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.8.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.8.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.8.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.8.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.8.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.9.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.9.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.9.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.9.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.9.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.9.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.9.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.9.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.10.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.10.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.10.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.10.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.10.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.10.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.10.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.10.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.layer_norm1.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.layer_norm1.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.self_attn.k_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.11.self_attn.k_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.self_attn.v_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.11.self_attn.v_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.self_attn.q_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.11.self_attn.q_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.self_attn.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.encoder.layers.11.self_attn.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.layer_norm2.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.layer_norm2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.encoder.layers.11.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.encoder.layers.11.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.encoder.layers.11.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.encoder.layers.11.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.post_layernorm.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.post_layernorm.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.head.probe: requires_grad=False, shape=torch.Size([1, 1, 768]), numel=768
  encoder.vision_model.head.attention.in_proj_weight: requires_grad=False, shape=torch.Size([2304, 768]), numel=1,769,472
  encoder.vision_model.head.attention.in_proj_bias: requires_grad=False, shape=torch.Size([2304]), numel=2,304
  encoder.vision_model.head.attention.out_proj.weight: requires_grad=False, shape=torch.Size([768, 768]), numel=589,824
  encoder.vision_model.head.attention.out_proj.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.head.layernorm.weight: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.head.layernorm.bias: requires_grad=False, shape=torch.Size([768]), numel=768
  encoder.vision_model.head.mlp.fc1.weight: requires_grad=False, shape=torch.Size([3072, 768]), numel=2,359,296
  encoder.vision_model.head.mlp.fc1.bias: requires_grad=False, shape=torch.Size([3072]), numel=3,072
  encoder.vision_model.head.mlp.fc2.weight: requires_grad=False, shape=torch.Size([768, 3072]), numel=2,359,296
  encoder.vision_model.head.mlp.fc2.bias: requires_grad=False, shape=torch.Size([768]), numel=768

参数统计:
  总参数数量: 3,426,594,816
  可训练参数数量: 29,491,200
  LoRA参数数量: 29,491,200
  连接器参数数量: 5,771,264
  可训练参数比例: 0.86%
==================================================

/home/jizhi/code/ad/models/MedM-VL/lvlm/train.py:167: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `LVLMTrainer.__init__`. Use `processing_class` instead.
  trainer = LVLMTrainer(
  0%|                                                                                                          | 0/2200 [00:00<?, ?it/s]/home/jizhi/anaconda3/envs/llm/lib/python3.10/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(

[shiym2000]

The warning “None of the inputs have requires_grad=True. Gradients will be None” does not affect normal training, as long as the training parameters are printed correctly.

It’s recommended to adjust the learning_rate and gradient_accumulation_steps and observe the training behavior over more epochs. In my local test running examples/finetune_2d.sh, everything worked normally.

******************************Stage 5******************************
Create trainer and train...
  0%|                                                                                                                                                                            | 0/2 [00:00<?, ?it/s]/yinghepool/shiym/miniconda/envs/medm/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/yinghepool/shiym/miniconda/envs/medm/lib/python3.10/site-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
/yinghepool/shiym/miniconda/envs/medm/lib/python3.10/site-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/yinghepool/shiym/miniconda/envs/medm/lib/python3.10/site-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
{'loss': 0.5833, 'grad_norm': 0.44699719548225403, 'learning_rate': 0.0002, 'epoch': 0.5}                                                                                                              
{'loss': 0.6748, 'grad_norm': 0.4990886151790619, 'learning_rate': 0.0, 'epoch': 1.0}                                                                                                                  
{'train_runtime': 4.3396, 'train_samples_per_second': 1.152, 'train_steps_per_second': 0.461, 'train_loss': 0.6290844976902008, 'epoch': 1.0}                                                          
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.02s/it]******************************Stage 6******************************
Save model...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.17s/it]
******************************Stage 6******************************
Save model...
[2025-10-17 09:23:46,713] [INFO] [launch.py:351:main] Process 5833 exits successfully.
[2025-10-17 09:23:54,722] [INFO] [launch.py:351:main] Process 5832 exits successfully.

[ctcnb]

好的，我会进行尝试，还有一个问题是，如果处理多个图片，是否需要添加多个\n<image>

[shiym2000]

In principle, it is not necessary.