cjkasbdkjnlakb/agent-0923-02

See axolotl config

axolotl version: 0.12.2

# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
# 是否以 8-bit 精度加载模型
load_in_8bit: false
# 是否以 4-bit 精度加载模型（与QLoRA绑定, 强制使用）
load_in_4bit: false
# 是否严格匹配模型结构，关闭表示可加载少部分差异结构（如以适配 adapter）
# strict: false
base_model: Qwen/Qwen3-4B-Instruct-2507
# 数据集设置
chat_template: qwen3
datasets:
  - path: /workspace/train_dir_0923-02/all_data.json # - 表示列表（list）中的一项, 即可以同时使用多个数据集
    type: chat_template # chat_template(自定义格式) alpaca
    roles_to_train: ["assistant"]
    field_messages: messages # 标识的字段
    message_property_mappings:  # message_property_mappings={'role':'role', 'content':'content'})
      role: role
      content: content
dataset_prepared_path:
val_set_size: 0.08
output_dir: /workspace/train_dir_0923-02/checkpoints/0923-02
sequence_len: 16384 # 模型所能处理的最大上下文长度(默认2048)
pad_to_sequence_len: true
# context_parallel_size: 2 # 长序列拆分至多个GPU(强制要求 mirco_batch_size: 1)
sample_packing: false # 在训练时将多个样本拼接（packing）成一个长序列(sequence_len)输入到模型中，以提高训练效率。
eval_sample_packing: false # 评估时拼接多个样本
# 训练超参数
adapter: lora  # lora qlora
lora_model_dir:
lora_r: 32 # lora_r默认首选 16，平衡精度与显存
lora_alpha: 32 # 缩放系数，用于控制 LoRA 的影响力, 一般设为 2*r 或 4*r
lora_dropout: 0.05    # 从0.05改为0.1，增加dropout
lora_target_linear: true
micro_batch_size: 4 # 微批次大小 94G的H100可以设为4(Token为1w)
gradient_accumulation_steps: 4 # 梯度累积: 将多个微批次的梯度(micro_batch_size)累积起来，然后更新模型权重 有效 Batch 常取 16: 小于 8 训练会抖，大于 32 只会更耗时、收益有限
auto_find_batch_size: false # 允许Axolotl不断调整batch_size  ⚠️Zero-3不适用
num_epochs: 3
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 5e-5
# bf16: auto + tf32: true，可获得更好的稳定性和性能。
bf16: auto
tf32: true
# early_stopping_patience:
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
# auto_resume_from_checkpoints: true #自动从output_dir寻找最新checkpoint断点恢复

logging_steps: 1
logging_dir: /workspace/train_dir_0923-02/logs

flash_attention: true
warmup_ratio: 0.03
evals_per_epoch: 8   # 增加评估频次，从4改为8
saves_per_epoch: 1   # 增加保存频次，便于选择最佳checkpoint
weight_decay: 0.01   # 从0.0改为0.01，增加正则化




fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_limit_all_gathers: true
  fsdp_sync_module_states: true
  fsdp_offload_params: false  # H200显存足够，无需offload
  fsdp_use_orig_params: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_transformer_layer_cls_to_wrap: Qwen3DecoderLayer
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sharding_strategy: FULL_SHARD

workspace/train_dir_0923-02/checkpoints/0923-02

This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 on the /workspace/train_dir_0923-02/all_data.json dataset. It achieves the following results on the evaluation set:

• Loss: 0.0369
• Memory/max Mem Active(gib): 129.15
• Memory/max Mem Allocated(gib): 128.95
• Memory/device Mem Reserved(gib): 130.58

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• total_eval_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 64
• training_steps: 2139

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	1.0423	98.3	98.1	99.56
0.0699	0.1263	90	0.0843	129.15	128.95	130.58
0.0667	0.2525	180	0.0614	129.15	128.95	130.58
0.0681	0.3788	270	0.0547	129.15	128.95	130.58
0.0523	0.5051	360	0.0508	129.15	128.95	130.58
0.0519	0.6314	450	0.0486	129.15	128.95	130.58
0.0502	0.7576	540	0.0471	129.15	128.95	130.58
0.0404	0.8839	630	0.0458	129.15	128.95	130.58
0.0489	1.0098	720	0.0442	129.15	128.95	130.58
0.0413	1.1361	810	0.0437	129.15	128.95	130.58
0.0514	1.2624	900	0.0426	129.15	128.95	130.58
0.0515	1.3886	990	0.0419	129.15	128.95	130.58
0.0436	1.5149	1080	0.0412	129.15	128.95	130.58
0.0449	1.6412	1170	0.0403	129.15	128.95	130.58
0.0464	1.7675	1260	0.0397	129.15	128.95	130.58
0.0487	1.8937	1350	0.0391	129.15	128.95	130.58
0.036	2.0196	1440	0.0386	129.15	128.95	130.58
0.0428	2.1459	1530	0.0382	129.15	128.95	130.58
0.043	2.2722	1620	0.0378	129.15	128.95	130.58
0.0397	2.3985	1710	0.0374	129.15	128.95	130.58
0.0409	2.5247	1800	0.0372	129.15	128.95	130.58
0.0317	2.6510	1890	0.0370	129.15	128.95	130.58
0.0362	2.7773	1980	0.0369	129.15	128.95	130.58
0.0397	2.9035	2070	0.0369	129.15	128.95	130.58

Framework versions

• PEFT 0.17.0
• Transformers 4.55.2
• Pytorch 2.6.0+cu126
• Datasets 4.0.0
• Tokenizers 0.21.4