[2025-07-28 15:49:28] Experiment directory created at ./lora_log/012-Wan2.1-1.3B-F81
[2025-07-28 15:49:28] Creating WanModel from ./Wan2.1-T2V-1.3B
[2025-07-28 15:49:28] Creating T5EncoderModel from ./Wan2.1-T2V-1.3B
[2025-07-28 15:50:36] loading ./Wan2.1-T2V-1.3B/models_t5_umt5-xxl-enc-bf16.pth
[2025-07-28 15:50:55] Creating WanVAE from ./Wan2.1-T2V-1.3B
[2025-07-28 15:50:55] loading ./Wan2.1-T2V-1.3B/Wan2.1_VAE.pth
[2025-07-28 15:51:13] Using ema ckpt!
[2025-07-28 15:51:14] Successfully Load 100.0% original pretrained model weights!
[2025-07-28 15:51:38] Successfully load model at ./lora_log/011-Wan2.1-1.3B-F81/checkpoints/0004416.pt!
[2025-07-28 15:52:11] Trainable Modules: 600
[2025-07-28 15:52:11] WARNING: Only train ('lora_up', 'lora_down') parametes!
[2025-07-28 15:52:11] Model Parameters: 1,506,487,360
[2025-07-28 15:52:11] Trainable Model Parameters: 87,490,560
[2025-07-28 15:52:43] Dataset contains: 251,859
[2025-07-28 15:52:43] Total train batch size (w. parallel, distributed & accumulation) = 256
[2025-07-28 16:16:34] (step=0004420/epoch=0000) Train Loss: 0.1715, Gradient Norm: 0.0074, Sec/Train Steps: 357.89, lr: 0.000100
[2025-07-28 16:36:02] (step=0004424/epoch=0000) Train Loss: 0.1736, Gradient Norm: 0.0142, Sec/Train Steps: 292.03, lr: 0.000100
[2025-07-28 16:55:31] (step=0004428/epoch=0000) Train Loss: 0.1690, Gradient Norm: 0.0190, Sec/Train Steps: 292.18, lr: 0.000100
[2025-07-28 17:14:57] (step=0004432/epoch=0000) Train Loss: 0.1672, Gradient Norm: 0.0141, Sec/Train Steps: 291.52, lr: 0.000100
[2025-07-28 17:34:24] (step=0004436/epoch=0000) Train Loss: 0.1713, Gradient Norm: 0.0065, Sec/Train Steps: 291.62, lr: 0.000100
[2025-07-28 17:53:53] (step=0004440/epoch=0000) Train Loss: 0.1715, Gradient Norm: 0.0085, Sec/Train Steps: 292.32, lr: 0.000100
[2025-07-28 18:13:25] (step=0004444/epoch=0000) Train Loss: 0.1677, Gradient Norm: 0.0078, Sec/Train Steps: 293.05, lr: 0.000100
[2025-07-28 18:32:50] (step=0004448/epoch=0000) Train Loss: 0.1704, Gradient Norm: 0.0064, Sec/Train Steps: 291.17, lr: 0.000100
[2025-07-28 18:52:15] (step=0004452/epoch=0000) Train Loss: 0.1704, Gradient Norm: 0.0056, Sec/Train Steps: 291.32, lr: 0.000100
[2025-07-28 19:11:43] (step=0004456/epoch=0000) Train Loss: 0.1656, Gradient Norm: 0.0099, Sec/Train Steps: 291.92, lr: 0.000100
[2025-07-28 19:31:09] (step=0004460/epoch=0000) Train Loss: 0.1673, Gradient Norm: 0.0063, Sec/Train Steps: 291.44, lr: 0.000100
[2025-07-28 19:50:38] (step=0004464/epoch=0000) Train Loss: 0.1723, Gradient Norm: 0.0090, Sec/Train Steps: 292.17, lr: 0.000100
[2025-07-28 19:50:40] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004464.pt
[2025-07-28 20:10:03] (step=0004468/epoch=0000) Train Loss: 0.1796, Gradient Norm: 0.0073, Sec/Train Steps: 291.37, lr: 0.000100
[2025-07-28 20:29:26] (step=0004472/epoch=0000) Train Loss: 0.1664, Gradient Norm: 0.0059, Sec/Train Steps: 290.83, lr: 0.000100
[2025-07-28 20:48:52] (step=0004476/epoch=0000) Train Loss: 0.1653, Gradient Norm: 0.0101, Sec/Train Steps: 291.44, lr: 0.000100
[2025-07-28 21:08:21] (step=0004480/epoch=0000) Train Loss: 0.1828, Gradient Norm: 0.0083, Sec/Train Steps: 292.30, lr: 0.000100
[2025-07-28 21:27:53] (step=0004484/epoch=0000) Train Loss: 0.1708, Gradient Norm: 0.0046, Sec/Train Steps: 293.03, lr: 0.000100
[2025-07-28 21:47:18] (step=0004488/epoch=0000) Train Loss: 0.1708, Gradient Norm: 0.0060, Sec/Train Steps: 291.18, lr: 0.000100
[2025-07-28 22:06:46] (step=0004492/epoch=0000) Train Loss: 0.1738, Gradient Norm: 0.0077, Sec/Train Steps: 291.98, lr: 0.000100
[2025-07-28 22:26:14] (step=0004496/epoch=0000) Train Loss: 0.1691, Gradient Norm: 0.0061, Sec/Train Steps: 291.99, lr: 0.000100
[2025-07-28 22:45:45] (step=0004500/epoch=0000) Train Loss: 0.1714, Gradient Norm: 0.0090, Sec/Train Steps: 292.80, lr: 0.000100
[2025-07-28 23:05:11] (step=0004504/epoch=0000) Train Loss: 0.1784, Gradient Norm: 0.0135, Sec/Train Steps: 291.38, lr: 0.000100
[2025-07-28 23:24:32] (step=0004508/epoch=0000) Train Loss: 0.1782, Gradient Norm: 0.0110, Sec/Train Steps: 290.17, lr: 0.000100
[2025-07-28 23:44:02] (step=0004512/epoch=0000) Train Loss: 0.1611, Gradient Norm: 0.0057, Sec/Train Steps: 292.48, lr: 0.000100
[2025-07-28 23:44:05] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004512.pt
[2025-07-29 00:03:35] (step=0004516/epoch=0000) Train Loss: 0.1687, Gradient Norm: 0.0098, Sec/Train Steps: 293.40, lr: 0.000100
[2025-07-29 00:23:03] (step=0004520/epoch=0000) Train Loss: 0.1673, Gradient Norm: 0.0063, Sec/Train Steps: 291.87, lr: 0.000100
[2025-07-29 00:42:28] (step=0004524/epoch=0000) Train Loss: 0.1674, Gradient Norm: 0.0062, Sec/Train Steps: 291.39, lr: 0.000100
[2025-07-29 01:01:56] (step=0004528/epoch=0000) Train Loss: 0.1704, Gradient Norm: 0.0053, Sec/Train Steps: 291.90, lr: 0.000100
[2025-07-29 01:21:24] (step=0004532/epoch=0000) Train Loss: 0.1679, Gradient Norm: 0.0050, Sec/Train Steps: 292.11, lr: 0.000100
[2025-07-29 01:40:51] (step=0004536/epoch=0000) Train Loss: 0.1680, Gradient Norm: 0.0071, Sec/Train Steps: 291.62, lr: 0.000100
[2025-07-29 02:00:18] (step=0004540/epoch=0000) Train Loss: 0.1739, Gradient Norm: 0.0110, Sec/Train Steps: 291.78, lr: 0.000100
[2025-07-29 02:19:46] (step=0004544/epoch=0000) Train Loss: 0.1708, Gradient Norm: 0.0056, Sec/Train Steps: 291.93, lr: 0.000100
[2025-07-29 02:39:14] (step=0004548/epoch=0000) Train Loss: 0.1724, Gradient Norm: 0.0074, Sec/Train Steps: 292.17, lr: 0.000100
[2025-07-29 02:58:41] (step=0004552/epoch=0000) Train Loss: 0.1651, Gradient Norm: 0.0060, Sec/Train Steps: 291.68, lr: 0.000100
[2025-07-29 03:18:09] (step=0004556/epoch=0000) Train Loss: 0.1769, Gradient Norm: 0.0088, Sec/Train Steps: 292.08, lr: 0.000100
[2025-07-29 03:37:39] (step=0004560/epoch=0000) Train Loss: 0.1754, Gradient Norm: 0.0110, Sec/Train Steps: 292.40, lr: 0.000100
[2025-07-29 03:37:43] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004560.pt
[2025-07-29 03:57:10] (step=0004564/epoch=0000) Train Loss: 0.1703, Gradient Norm: 0.0060, Sec/Train Steps: 292.85, lr: 0.000100
[2025-07-29 04:16:35] (step=0004568/epoch=0000) Train Loss: 0.1671, Gradient Norm: 0.0057, Sec/Train Steps: 291.06, lr: 0.000100
[2025-07-29 04:36:06] (step=0004572/epoch=0000) Train Loss: 0.1784, Gradient Norm: 0.0122, Sec/Train Steps: 292.91, lr: 0.000100
[2025-07-29 04:55:34] (step=0004576/epoch=0000) Train Loss: 0.1690, Gradient Norm: 0.0063, Sec/Train Steps: 291.97, lr: 0.000100
[2025-07-29 05:14:59] (step=0004580/epoch=0000) Train Loss: 0.1686, Gradient Norm: 0.0052, Sec/Train Steps: 291.33, lr: 0.000100
[2025-07-29 05:34:24] (step=0004584/epoch=0000) Train Loss: 0.1709, Gradient Norm: 0.0070, Sec/Train Steps: 291.18, lr: 0.000100
[2025-07-29 05:53:50] (step=0004588/epoch=0000) Train Loss: 0.1712, Gradient Norm: 0.0098, Sec/Train Steps: 291.42, lr: 0.000100
[2025-07-29 06:13:14] (step=0004592/epoch=0000) Train Loss: 0.1744, Gradient Norm: 0.0049, Sec/Train Steps: 291.01, lr: 0.000100
[2025-07-29 06:32:41] (step=0004596/epoch=0000) Train Loss: 0.1773, Gradient Norm: 0.0122, Sec/Train Steps: 291.45, lr: 0.000100
[2025-07-29 06:52:06] (step=0004600/epoch=0000) Train Loss: 0.1693, Gradient Norm: 0.0076, Sec/Train Steps: 291.27, lr: 0.000100
[2025-07-29 07:11:31] (step=0004604/epoch=0000) Train Loss: 0.1665, Gradient Norm: 0.0053, Sec/Train Steps: 291.23, lr: 0.000100
[2025-07-29 07:30:57] (step=0004608/epoch=0000) Train Loss: 0.1679, Gradient Norm: 0.0060, Sec/Train Steps: 291.52, lr: 0.000100
[2025-07-29 07:31:00] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004608.pt
[2025-07-29 07:50:29] (step=0004612/epoch=0000) Train Loss: 0.1713, Gradient Norm: 0.0063, Sec/Train Steps: 293.01, lr: 0.000100
[2025-07-29 08:10:00] (step=0004616/epoch=0000) Train Loss: 0.1676, Gradient Norm: 0.0070, Sec/Train Steps: 292.71, lr: 0.000100
[2025-07-29 08:29:27] (step=0004620/epoch=0000) Train Loss: 0.1658, Gradient Norm: 0.0047, Sec/Train Steps: 291.62, lr: 0.000100
[2025-07-29 08:48:51] (step=0004624/epoch=0000) Train Loss: 0.1732, Gradient Norm: 0.0081, Sec/Train Steps: 291.03, lr: 0.000100
[2025-07-29 09:08:19] (step=0004628/epoch=0000) Train Loss: 0.1679, Gradient Norm: 0.0058, Sec/Train Steps: 292.12, lr: 0.000100
[2025-07-29 09:27:49] (step=0004632/epoch=0000) Train Loss: 0.1764, Gradient Norm: 0.0067, Sec/Train Steps: 292.25, lr: 0.000100
[2025-07-29 09:47:14] (step=0004636/epoch=0000) Train Loss: 0.1700, Gradient Norm: 0.0061, Sec/Train Steps: 291.46, lr: 0.000100
[2025-07-29 10:06:39] (step=0004640/epoch=0000) Train Loss: 0.1761, Gradient Norm: 0.0073, Sec/Train Steps: 291.25, lr: 0.000100
[2025-07-29 10:26:10] (step=0004644/epoch=0000) Train Loss: 0.1752, Gradient Norm: 0.0073, Sec/Train Steps: 292.60, lr: 0.000100
[2025-07-29 10:45:38] (step=0004648/epoch=0000) Train Loss: 0.1686, Gradient Norm: 0.0038, Sec/Train Steps: 291.95, lr: 0.000100
[2025-07-29 11:05:02] (step=0004652/epoch=0000) Train Loss: 0.1658, Gradient Norm: 0.0061, Sec/Train Steps: 291.00, lr: 0.000100
[2025-07-29 11:24:30] (step=0004656/epoch=0000) Train Loss: 0.1671, Gradient Norm: 0.0049, Sec/Train Steps: 291.77, lr: 0.000100
[2025-07-29 11:24:33] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004656.pt
[2025-07-29 11:44:01] (step=0004660/epoch=0000) Train Loss: 0.1663, Gradient Norm: 0.0063, Sec/Train Steps: 292.75, lr: 0.000100
[2025-07-29 12:03:24] (step=0004664/epoch=0000) Train Loss: 0.1764, Gradient Norm: 0.0047, Sec/Train Steps: 290.84, lr: 0.000100
[2025-07-29 12:22:49] (step=0004668/epoch=0000) Train Loss: 0.1676, Gradient Norm: 0.0039, Sec/Train Steps: 291.18, lr: 0.000100
[2025-07-29 12:42:16] (step=0004672/epoch=0000) Train Loss: 0.1735, Gradient Norm: 0.0052, Sec/Train Steps: 291.95, lr: 0.000100
[2025-07-29 13:01:44] (step=0004676/epoch=0000) Train Loss: 0.1675, Gradient Norm: 0.0054, Sec/Train Steps: 291.96, lr: 0.000100
[2025-07-29 13:21:09] (step=0004680/epoch=0000) Train Loss: 0.1679, Gradient Norm: 0.0046, Sec/Train Steps: 291.08, lr: 0.000100
[2025-07-29 13:40:37] (step=0004684/epoch=0000) Train Loss: 0.1707, Gradient Norm: 0.0045, Sec/Train Steps: 292.12, lr: 0.000100
[2025-07-29 14:00:02] (step=0004688/epoch=0000) Train Loss: 0.1737, Gradient Norm: 0.0052, Sec/Train Steps: 291.18, lr: 0.000100
[2025-07-29 14:19:28] (step=0004692/epoch=0000) Train Loss: 0.1683, Gradient Norm: 0.0082, Sec/Train Steps: 291.44, lr: 0.000100
[2025-07-29 14:38:54] (step=0004696/epoch=0000) Train Loss: 0.1772, Gradient Norm: 0.0100, Sec/Train Steps: 291.53, lr: 0.000100
[2025-07-29 14:58:29] (step=0004700/epoch=0000) Train Loss: 0.1743, Gradient Norm: 0.0089, Sec/Train Steps: 293.82, lr: 0.000100
[2025-07-29 15:17:56] (step=0004704/epoch=0000) Train Loss: 0.1744, Gradient Norm: 0.0083, Sec/Train Steps: 291.70, lr: 0.000100
[2025-07-29 15:17:59] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004704.pt
[2025-07-29 15:37:24] (step=0004708/epoch=0000) Train Loss: 0.1767, Gradient Norm: 0.0084, Sec/Train Steps: 292.13, lr: 0.000100
[2025-07-29 15:56:54] (step=0004712/epoch=0000) Train Loss: 0.1752, Gradient Norm: 0.0077, Sec/Train Steps: 292.35, lr: 0.000100
[2025-07-29 16:16:17] (step=0004716/epoch=0000) Train Loss: 0.1723, Gradient Norm: 0.0061, Sec/Train Steps: 290.79, lr: 0.000100
[2025-07-29 16:35:43] (step=0004720/epoch=0000) Train Loss: 0.1672, Gradient Norm: 0.0075, Sec/Train Steps: 291.55, lr: 0.000100
[2025-07-29 16:55:09] (step=0004724/epoch=0000) Train Loss: 0.1770, Gradient Norm: 0.0134, Sec/Train Steps: 291.48, lr: 0.000100
[2025-07-29 17:14:34] (step=0004728/epoch=0000) Train Loss: 0.1699, Gradient Norm: 0.0050, Sec/Train Steps: 291.36, lr: 0.000100
[2025-07-29 17:33:58] (step=0004732/epoch=0000) Train Loss: 0.1674, Gradient Norm: 0.0053, Sec/Train Steps: 290.90, lr: 0.000100
[2025-07-29 17:53:28] (step=0004736/epoch=0000) Train Loss: 0.1728, Gradient Norm: 0.0072, Sec/Train Steps: 292.52, lr: 0.000100
[2025-07-29 18:12:57] (step=0004740/epoch=0000) Train Loss: 0.1668, Gradient Norm: 0.0050, Sec/Train Steps: 292.13, lr: 0.000100
[2025-07-29 18:32:23] (step=0004744/epoch=0000) Train Loss: 0.1752, Gradient Norm: 0.0103, Sec/Train Steps: 291.58, lr: 0.000100
[2025-07-29 18:51:51] (step=0004748/epoch=0000) Train Loss: 0.1693, Gradient Norm: 0.0085, Sec/Train Steps: 291.95, lr: 0.000100
[2025-07-29 19:11:18] (step=0004752/epoch=0000) Train Loss: 0.1682, Gradient Norm: 0.0066, Sec/Train Steps: 291.84, lr: 0.000100
[2025-07-29 19:11:21] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004752.pt
[2025-07-29 19:30:51] (step=0004756/epoch=0000) Train Loss: 0.1745, Gradient Norm: 0.0067, Sec/Train Steps: 293.10, lr: 0.000100
[2025-07-29 19:50:16] (step=0004760/epoch=0000) Train Loss: 0.1656, Gradient Norm: 0.0071, Sec/Train Steps: 291.39, lr: 0.000100
[2025-07-29 20:09:47] (step=0004764/epoch=0000) Train Loss: 0.1780, Gradient Norm: 0.0080, Sec/Train Steps: 292.73, lr: 0.000100
[2025-07-29 20:29:10] (step=0004768/epoch=0000) Train Loss: 0.1679, Gradient Norm: 0.0058, Sec/Train Steps: 290.79, lr: 0.000100
[2025-07-29 20:48:36] (step=0004772/epoch=0000) Train Loss: 0.1728, Gradient Norm: 0.0229, Sec/Train Steps: 291.33, lr: 0.000100
[2025-07-29 21:07:59] (step=0004776/epoch=0000) Train Loss: 0.1707, Gradient Norm: 0.0131, Sec/Train Steps: 290.80, lr: 0.000100
[2025-07-29 21:27:26] (step=0004780/epoch=0000) Train Loss: 0.1727, Gradient Norm: 0.0136, Sec/Train Steps: 291.68, lr: 0.000100
[2025-07-29 21:46:52] (step=0004784/epoch=0000) Train Loss: 0.1757, Gradient Norm: 0.0077, Sec/Train Steps: 291.59, lr: 0.000100
[2025-07-29 22:06:20] (step=0004788/epoch=0000) Train Loss: 0.1707, Gradient Norm: 0.0072, Sec/Train Steps: 291.92, lr: 0.000100
[2025-07-29 22:25:46] (step=0004792/epoch=0000) Train Loss: 0.1697, Gradient Norm: 0.0057, Sec/Train Steps: 291.39, lr: 0.000100
[2025-07-29 22:45:10] (step=0004796/epoch=0000) Train Loss: 0.1672, Gradient Norm: 0.0065, Sec/Train Steps: 291.01, lr: 0.000100
[2025-07-29 23:04:36] (step=0004800/epoch=0000) Train Loss: 0.1730, Gradient Norm: 0.0062, Sec/Train Steps: 291.35, lr: 0.000100
[2025-07-29 23:04:40] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004800.pt
[2025-07-29 23:24:05] (step=0004804/epoch=0000) Train Loss: 0.1639, Gradient Norm: 0.0052, Sec/Train Steps: 292.30, lr: 0.000100
[2025-07-29 23:43:29] (step=0004808/epoch=0000) Train Loss: 0.1689, Gradient Norm: 0.0072, Sec/Train Steps: 291.02, lr: 0.000100
[2025-07-30 00:02:54] (step=0004812/epoch=0000) Train Loss: 0.1818, Gradient Norm: 0.0115, Sec/Train Steps: 290.96, lr: 0.000100
[2025-07-30 00:22:18] (step=0004816/epoch=0000) Train Loss: 0.1653, Gradient Norm: 0.0113, Sec/Train Steps: 291.09, lr: 0.000100
[2025-07-30 00:41:45] (step=0004820/epoch=0000) Train Loss: 0.1668, Gradient Norm: 0.0078, Sec/Train Steps: 291.73, lr: 0.000100
[2025-07-30 01:01:10] (step=0004824/epoch=0000) Train Loss: 0.1761, Gradient Norm: 0.0084, Sec/Train Steps: 291.09, lr: 0.000100
[2025-07-30 01:20:39] (step=0004828/epoch=0000) Train Loss: 0.1740, Gradient Norm: 0.0050, Sec/Train Steps: 292.43, lr: 0.000100
[2025-07-30 01:40:08] (step=0004832/epoch=0000) Train Loss: 0.1755, Gradient Norm: 0.0076, Sec/Train Steps: 292.25, lr: 0.000100
[2025-07-30 01:59:33] (step=0004836/epoch=0000) Train Loss: 0.1758, Gradient Norm: 0.0062, Sec/Train Steps: 291.19, lr: 0.000100
[2025-07-30 02:19:03] (step=0004840/epoch=0000) Train Loss: 0.1777, Gradient Norm: 0.0145, Sec/Train Steps: 292.52, lr: 0.000100
[2025-07-30 02:38:32] (step=0004844/epoch=0000) Train Loss: 0.1678, Gradient Norm: 0.0084, Sec/Train Steps: 291.96, lr: 0.000100
[2025-07-30 02:58:01] (step=0004848/epoch=0000) Train Loss: 0.1697, Gradient Norm: 0.0095, Sec/Train Steps: 292.42, lr: 0.000100
[2025-07-30 02:58:05] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004848.pt
[2025-07-30 03:17:28] (step=0004852/epoch=0000) Train Loss: 0.1804, Gradient Norm: 0.0059, Sec/Train Steps: 291.73, lr: 0.000100
[2025-07-30 03:36:55] (step=0004856/epoch=0000) Train Loss: 0.1672, Gradient Norm: 0.0065, Sec/Train Steps: 291.56, lr: 0.000100
[2025-07-30 03:56:20] (step=0004860/epoch=0000) Train Loss: 0.1708, Gradient Norm: 0.0085, Sec/Train Steps: 291.38, lr: 0.000100
[2025-07-30 04:15:45] (step=0004864/epoch=0000) Train Loss: 0.1727, Gradient Norm: 0.0072, Sec/Train Steps: 291.13, lr: 0.000100
[2025-07-30 04:35:08] (step=0004868/epoch=0000) Train Loss: 0.1658, Gradient Norm: 0.0067, Sec/Train Steps: 290.87, lr: 0.000100
[2025-07-30 04:54:36] (step=0004872/epoch=0000) Train Loss: 0.1681, Gradient Norm: 0.0044, Sec/Train Steps: 291.81, lr: 0.000100
[2025-07-30 05:13:56] (step=0004876/epoch=0000) Train Loss: 0.1738, Gradient Norm: 0.0060, Sec/Train Steps: 290.12, lr: 0.000100
[2025-07-30 05:33:24] (step=0004880/epoch=0000) Train Loss: 0.1766, Gradient Norm: 0.0072, Sec/Train Steps: 291.84, lr: 0.000100
[2025-07-30 05:52:48] (step=0004884/epoch=0000) Train Loss: 0.1747, Gradient Norm: 0.0053, Sec/Train Steps: 291.15, lr: 0.000100
[2025-07-30 06:12:13] (step=0004888/epoch=0000) Train Loss: 0.1717, Gradient Norm: 0.0078, Sec/Train Steps: 291.32, lr: 0.000100
[2025-07-30 06:31:38] (step=0004892/epoch=0000) Train Loss: 0.1798, Gradient Norm: 0.0062, Sec/Train Steps: 291.15, lr: 0.000100
[2025-07-30 06:51:04] (step=0004896/epoch=0000) Train Loss: 0.1661, Gradient Norm: 0.0114, Sec/Train Steps: 291.42, lr: 0.000100
[2025-07-30 06:51:06] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004896.pt
[2025-07-30 07:10:34] (step=0004900/epoch=0000) Train Loss: 0.1737, Gradient Norm: 0.0110, Sec/Train Steps: 292.52, lr: 0.000100
[2025-07-30 07:30:01] (step=0004904/epoch=0000) Train Loss: 0.1634, Gradient Norm: 0.0070, Sec/Train Steps: 291.73, lr: 0.000100
[2025-07-30 07:49:30] (step=0004908/epoch=0000) Train Loss: 0.1717, Gradient Norm: 0.0088, Sec/Train Steps: 292.35, lr: 0.000100
[2025-07-30 08:09:06] (step=0004912/epoch=0000) Train Loss: 0.1702, Gradient Norm: 0.0057, Sec/Train Steps: 294.01, lr: 0.000100
[2025-07-30 08:28:29] (step=0004916/epoch=0000) Train Loss: 0.1666, Gradient Norm: 0.0072, Sec/Train Steps: 290.74, lr: 0.000100
[2025-07-30 08:47:57] (step=0004920/epoch=0000) Train Loss: 0.1751, Gradient Norm: 0.0059, Sec/Train Steps: 291.95, lr: 0.000100
[2025-07-30 09:07:24] (step=0004924/epoch=0000) Train Loss: 0.1677, Gradient Norm: 0.0089, Sec/Train Steps: 291.60, lr: 0.000100
[2025-07-30 09:26:52] (step=0004928/epoch=0000) Train Loss: 0.1657, Gradient Norm: 0.0078, Sec/Train Steps: 292.15, lr: 0.000100
[2025-07-30 09:46:18] (step=0004932/epoch=0000) Train Loss: 0.1747, Gradient Norm: 0.0053, Sec/Train Steps: 291.54, lr: 0.000100
[2025-07-30 10:05:43] (step=0004936/epoch=0000) Train Loss: 0.1728, Gradient Norm: 0.0090, Sec/Train Steps: 291.07, lr: 0.000100
[2025-07-30 10:25:08] (step=0004940/epoch=0000) Train Loss: 0.1716, Gradient Norm: 0.0107, Sec/Train Steps: 291.40, lr: 0.000100
[2025-07-30 10:44:35] (step=0004944/epoch=0000) Train Loss: 0.1767, Gradient Norm: 0.0073, Sec/Train Steps: 291.63, lr: 0.000100
[2025-07-30 10:44:38] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004944.pt
[2025-07-30 11:04:04] (step=0004948/epoch=0000) Train Loss: 0.1690, Gradient Norm: 0.0103, Sec/Train Steps: 292.40, lr: 0.000100
[2025-07-30 11:23:30] (step=0004952/epoch=0000) Train Loss: 0.1715, Gradient Norm: 0.0085, Sec/Train Steps: 291.47, lr: 0.000100
[2025-07-30 11:42:58] (step=0004956/epoch=0000) Train Loss: 0.1707, Gradient Norm: 0.0115, Sec/Train Steps: 291.87, lr: 0.000100
[2025-07-30 12:02:28] (step=0004960/epoch=0000) Train Loss: 0.1734, Gradient Norm: 0.0078, Sec/Train Steps: 292.52, lr: 0.000100
[2025-07-30 12:21:55] (step=0004964/epoch=0000) Train Loss: 0.1658, Gradient Norm: 0.0087, Sec/Train Steps: 291.70, lr: 0.000100
[2025-07-30 12:41:21] (step=0004968/epoch=0000) Train Loss: 0.1713, Gradient Norm: 0.0071, Sec/Train Steps: 291.53, lr: 0.000100
[2025-07-30 13:00:44] (step=0004972/epoch=0000) Train Loss: 0.1723, Gradient Norm: 0.0072, Sec/Train Steps: 290.80, lr: 0.000100
[2025-07-30 13:20:08] (step=0004976/epoch=0000) Train Loss: 0.1690, Gradient Norm: 0.0071, Sec/Train Steps: 290.84, lr: 0.000100
[2025-07-30 13:39:33] (step=0004980/epoch=0000) Train Loss: 0.1778, Gradient Norm: 0.0073, Sec/Train Steps: 291.24, lr: 0.000100
[2025-07-30 13:58:58] (step=0004984/epoch=0000) Train Loss: 0.1719, Gradient Norm: 0.0087, Sec/Train Steps: 291.30, lr: 0.000100
[2025-07-30 14:18:31] (step=0004988/epoch=0000) Train Loss: 0.1744, Gradient Norm: 0.0054, Sec/Train Steps: 293.16, lr: 0.000100
[2025-07-30 14:37:59] (step=0004992/epoch=0000) Train Loss: 0.1759, Gradient Norm: 0.0068, Sec/Train Steps: 291.88, lr: 0.000100
[2025-07-30 14:38:03] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0004992.pt
[2025-07-30 14:57:30] (step=0004996/epoch=0000) Train Loss: 0.1701, Gradient Norm: 0.0063, Sec/Train Steps: 292.79, lr: 0.000100
[2025-07-30 15:16:57] (step=0005000/epoch=0000) Train Loss: 0.1736, Gradient Norm: 0.0069, Sec/Train Steps: 291.80, lr: 0.000100
[2025-07-30 15:36:23] (step=0005004/epoch=0000) Train Loss: 0.1686, Gradient Norm: 0.0083, Sec/Train Steps: 291.41, lr: 0.000100
[2025-07-30 15:55:46] (step=0005008/epoch=0000) Train Loss: 0.1713, Gradient Norm: 0.0108, Sec/Train Steps: 290.57, lr: 0.000100
[2025-07-30 16:15:14] (step=0005012/epoch=0000) Train Loss: 0.1643, Gradient Norm: 0.0077, Sec/Train Steps: 291.93, lr: 0.000100
[2025-07-30 16:34:40] (step=0005016/epoch=0000) Train Loss: 0.1729, Gradient Norm: 0.0074, Sec/Train Steps: 291.59, lr: 0.000100
[2025-07-30 16:54:07] (step=0005020/epoch=0000) Train Loss: 0.1676, Gradient Norm: 0.0099, Sec/Train Steps: 291.76, lr: 0.000100
[2025-07-30 17:13:34] (step=0005024/epoch=0000) Train Loss: 0.1704, Gradient Norm: 0.0087, Sec/Train Steps: 291.73, lr: 0.000100
[2025-07-30 17:33:01] (step=0005028/epoch=0000) Train Loss: 0.1752, Gradient Norm: 0.0045, Sec/Train Steps: 291.64, lr: 0.000100
[2025-07-30 17:52:30] (step=0005032/epoch=0000) Train Loss: 0.1709, Gradient Norm: 0.0059, Sec/Train Steps: 292.30, lr: 0.000100
[2025-07-30 18:11:55] (step=0005036/epoch=0000) Train Loss: 0.1701, Gradient Norm: 0.0205, Sec/Train Steps: 291.25, lr: 0.000100
[2025-07-30 18:31:21] (step=0005040/epoch=0000) Train Loss: 0.1678, Gradient Norm: 0.0128, Sec/Train Steps: 291.23, lr: 0.000100
[2025-07-30 18:31:24] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005040.pt
[2025-07-30 18:50:55] (step=0005044/epoch=0000) Train Loss: 0.1740, Gradient Norm: 0.0096, Sec/Train Steps: 293.51, lr: 0.000100
[2025-07-30 19:10:27] (step=0005048/epoch=0000) Train Loss: 0.1675, Gradient Norm: 0.0061, Sec/Train Steps: 292.96, lr: 0.000100
[2025-07-30 19:29:55] (step=0005052/epoch=0000) Train Loss: 0.1683, Gradient Norm: 0.0088, Sec/Train Steps: 292.10, lr: 0.000100
[2025-07-30 19:49:23] (step=0005056/epoch=0000) Train Loss: 0.1712, Gradient Norm: 0.0072, Sec/Train Steps: 292.07, lr: 0.000100
[2025-07-30 20:08:48] (step=0005060/epoch=0000) Train Loss: 0.1674, Gradient Norm: 0.0066, Sec/Train Steps: 290.93, lr: 0.000100
[2025-07-30 20:28:18] (step=0005064/epoch=0000) Train Loss: 0.1757, Gradient Norm: 0.0060, Sec/Train Steps: 292.58, lr: 0.000100
[2025-07-30 20:47:42] (step=0005068/epoch=0000) Train Loss: 0.1640, Gradient Norm: 0.0055, Sec/Train Steps: 290.97, lr: 0.000100
[2025-07-30 21:07:09] (step=0005072/epoch=0000) Train Loss: 0.1736, Gradient Norm: 0.0079, Sec/Train Steps: 291.60, lr: 0.000100
[2025-07-30 21:26:33] (step=0005076/epoch=0000) Train Loss: 0.1662, Gradient Norm: 0.0081, Sec/Train Steps: 291.06, lr: 0.000100
[2025-07-30 21:46:01] (step=0005080/epoch=0000) Train Loss: 0.1719, Gradient Norm: 0.0104, Sec/Train Steps: 292.10, lr: 0.000100
[2025-07-30 22:05:25] (step=0005084/epoch=0000) Train Loss: 0.1759, Gradient Norm: 0.0138, Sec/Train Steps: 291.06, lr: 0.000100
[2025-07-30 22:24:52] (step=0005088/epoch=0000) Train Loss: 0.1731, Gradient Norm: 0.0080, Sec/Train Steps: 291.71, lr: 0.000100
[2025-07-30 22:24:56] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005088.pt
[2025-07-30 22:44:21] (step=0005092/epoch=0000) Train Loss: 0.1701, Gradient Norm: 0.0062, Sec/Train Steps: 292.08, lr: 0.000100
[2025-07-30 23:03:51] (step=0005096/epoch=0000) Train Loss: 0.1728, Gradient Norm: 0.0070, Sec/Train Steps: 292.43, lr: 0.000100
[2025-07-30 23:23:14] (step=0005100/epoch=0000) Train Loss: 0.1731, Gradient Norm: 0.0074, Sec/Train Steps: 290.73, lr: 0.000100
[2025-07-30 23:42:41] (step=0005104/epoch=0000) Train Loss: 0.1721, Gradient Norm: 0.0092, Sec/Train Steps: 291.80, lr: 0.000100
[2025-07-31 00:02:06] (step=0005108/epoch=0000) Train Loss: 0.1698, Gradient Norm: 0.0084, Sec/Train Steps: 291.19, lr: 0.000100
[2025-07-31 00:21:32] (step=0005112/epoch=0000) Train Loss: 0.1708, Gradient Norm: 0.0074, Sec/Train Steps: 291.47, lr: 0.000100
[2025-07-31 00:40:58] (step=0005116/epoch=0000) Train Loss: 0.1658, Gradient Norm: 0.0070, Sec/Train Steps: 291.46, lr: 0.000100
[2025-07-31 01:00:26] (step=0005120/epoch=0000) Train Loss: 0.1683, Gradient Norm: 0.0097, Sec/Train Steps: 292.14, lr: 0.000100
[2025-07-31 01:19:55] (step=0005124/epoch=0000) Train Loss: 0.1733, Gradient Norm: 0.0082, Sec/Train Steps: 292.25, lr: 0.000100
[2025-07-31 01:39:21] (step=0005128/epoch=0000) Train Loss: 0.1701, Gradient Norm: 0.0089, Sec/Train Steps: 291.55, lr: 0.000100
[2025-07-31 01:58:49] (step=0005132/epoch=0000) Train Loss: 0.1705, Gradient Norm: 0.0087, Sec/Train Steps: 291.94, lr: 0.000100
[2025-07-31 02:18:16] (step=0005136/epoch=0000) Train Loss: 0.1740, Gradient Norm: 0.0134, Sec/Train Steps: 291.73, lr: 0.000100
[2025-07-31 02:18:19] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005136.pt
[2025-07-31 02:37:45] (step=0005140/epoch=0000) Train Loss: 0.1695, Gradient Norm: 0.0074, Sec/Train Steps: 292.13, lr: 0.000100
[2025-07-31 02:57:12] (step=0005144/epoch=0000) Train Loss: 0.1737, Gradient Norm: 0.0093, Sec/Train Steps: 291.88, lr: 0.000100
[2025-07-31 03:16:44] (step=0005148/epoch=0000) Train Loss: 0.1701, Gradient Norm: 0.0100, Sec/Train Steps: 292.90, lr: 0.000100
[2025-07-31 03:36:13] (step=0005152/epoch=0000) Train Loss: 0.1669, Gradient Norm: 0.0068, Sec/Train Steps: 292.16, lr: 0.000100
[2025-07-31 03:55:40] (step=0005156/epoch=0000) Train Loss: 0.1732, Gradient Norm: 0.0068, Sec/Train Steps: 291.90, lr: 0.000100
[2025-07-31 04:15:10] (step=0005160/epoch=0000) Train Loss: 0.1750, Gradient Norm: 0.0075, Sec/Train Steps: 292.30, lr: 0.000100
[2025-07-31 04:34:36] (step=0005164/epoch=0000) Train Loss: 0.1646, Gradient Norm: 0.0076, Sec/Train Steps: 291.69, lr: 0.000100
[2025-07-31 04:54:06] (step=0005168/epoch=0000) Train Loss: 0.1737, Gradient Norm: 0.0074, Sec/Train Steps: 292.33, lr: 0.000100
[2025-07-31 05:13:32] (step=0005172/epoch=0000) Train Loss: 0.1738, Gradient Norm: 0.0065, Sec/Train Steps: 291.60, lr: 0.000100
[2025-07-31 05:32:59] (step=0005176/epoch=0000) Train Loss: 0.1636, Gradient Norm: 0.0061, Sec/Train Steps: 291.83, lr: 0.000100
[2025-07-31 05:52:24] (step=0005180/epoch=0000) Train Loss: 0.1758, Gradient Norm: 0.0063, Sec/Train Steps: 291.22, lr: 0.000100
[2025-07-31 06:11:51] (step=0005184/epoch=0000) Train Loss: 0.1685, Gradient Norm: 0.0085, Sec/Train Steps: 291.41, lr: 0.000100
[2025-07-31 06:11:54] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005184.pt
[2025-07-31 06:31:20] (step=0005188/epoch=0000) Train Loss: 0.1706, Gradient Norm: 0.0052, Sec/Train Steps: 292.31, lr: 0.000100
[2025-07-31 06:50:48] (step=0005192/epoch=0000) Train Loss: 0.1763, Gradient Norm: 0.0055, Sec/Train Steps: 291.88, lr: 0.000100
[2025-07-31 07:10:11] (step=0005196/epoch=0000) Train Loss: 0.1703, Gradient Norm: 0.0066, Sec/Train Steps: 290.98, lr: 0.000100
[2025-07-31 07:29:39] (step=0005200/epoch=0000) Train Loss: 0.1663, Gradient Norm: 0.0063, Sec/Train Steps: 291.86, lr: 0.000100
[2025-07-31 07:49:09] (step=0005204/epoch=0000) Train Loss: 0.1728, Gradient Norm: 0.0115, Sec/Train Steps: 292.28, lr: 0.000100
[2025-07-31 08:08:36] (step=0005208/epoch=0000) Train Loss: 0.1755, Gradient Norm: 0.0084, Sec/Train Steps: 291.68, lr: 0.000100
[2025-07-31 08:28:02] (step=0005212/epoch=0000) Train Loss: 0.1665, Gradient Norm: 0.0049, Sec/Train Steps: 291.47, lr: 0.000100
[2025-07-31 08:47:31] (step=0005216/epoch=0000) Train Loss: 0.1672, Gradient Norm: 0.0050, Sec/Train Steps: 292.19, lr: 0.000100
[2025-07-31 09:06:57] (step=0005220/epoch=0000) Train Loss: 0.1732, Gradient Norm: 0.0067, Sec/Train Steps: 291.71, lr: 0.000100
[2025-07-31 09:26:25] (step=0005224/epoch=0000) Train Loss: 0.1735, Gradient Norm: 0.0055, Sec/Train Steps: 291.75, lr: 0.000100
[2025-07-31 09:45:51] (step=0005228/epoch=0000) Train Loss: 0.1673, Gradient Norm: 0.0122, Sec/Train Steps: 291.56, lr: 0.000100
[2025-07-31 10:05:21] (step=0005232/epoch=0000) Train Loss: 0.1687, Gradient Norm: 0.0072, Sec/Train Steps: 292.30, lr: 0.000100
[2025-07-31 10:05:23] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005232.pt
[2025-07-31 10:24:50] (step=0005236/epoch=0000) Train Loss: 0.1719, Gradient Norm: 0.0092, Sec/Train Steps: 292.35, lr: 0.000100
[2025-07-31 10:44:19] (step=0005240/epoch=0000) Train Loss: 0.1777, Gradient Norm: 0.0050, Sec/Train Steps: 292.24, lr: 0.000100
[2025-07-31 11:03:45] (step=0005244/epoch=0000) Train Loss: 0.1672, Gradient Norm: 0.0090, Sec/Train Steps: 291.54, lr: 0.000100
[2025-07-31 11:23:18] (step=0005248/epoch=0000) Train Loss: 0.1639, Gradient Norm: 0.0064, Sec/Train Steps: 293.10, lr: 0.000100
[2025-07-31 11:42:45] (step=0005252/epoch=0000) Train Loss: 0.1769, Gradient Norm: 0.0065, Sec/Train Steps: 291.86, lr: 0.000100
[2025-07-31 12:02:13] (step=0005256/epoch=0000) Train Loss: 0.1697, Gradient Norm: 0.0064, Sec/Train Steps: 291.66, lr: 0.000100
[2025-07-31 12:21:35] (step=0005260/epoch=0000) Train Loss: 0.1629, Gradient Norm: 0.0060, Sec/Train Steps: 290.64, lr: 0.000100
[2025-07-31 12:41:03] (step=0005264/epoch=0000) Train Loss: 0.1622, Gradient Norm: 0.0059, Sec/Train Steps: 291.95, lr: 0.000100
[2025-07-31 13:00:33] (step=0005268/epoch=0000) Train Loss: 0.1717, Gradient Norm: 0.0055, Sec/Train Steps: 292.55, lr: 0.000100
[2025-07-31 13:19:57] (step=0005272/epoch=0000) Train Loss: 0.1746, Gradient Norm: 0.0079, Sec/Train Steps: 290.94, lr: 0.000100
[2025-07-31 13:39:26] (step=0005276/epoch=0000) Train Loss: 0.1625, Gradient Norm: 0.0071, Sec/Train Steps: 292.29, lr: 0.000100
[2025-07-31 13:58:50] (step=0005280/epoch=0000) Train Loss: 0.1701, Gradient Norm: 0.0061, Sec/Train Steps: 291.02, lr: 0.000100
[2025-07-31 13:58:53] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005280.pt
[2025-07-31 14:18:23] (step=0005284/epoch=0000) Train Loss: 0.1761, Gradient Norm: 0.0062, Sec/Train Steps: 293.14, lr: 0.000100
[2025-07-31 14:37:48] (step=0005288/epoch=0000) Train Loss: 0.1697, Gradient Norm: 0.0071, Sec/Train Steps: 291.26, lr: 0.000100
[2025-07-31 14:57:16] (step=0005292/epoch=0000) Train Loss: 0.1769, Gradient Norm: 0.0086, Sec/Train Steps: 292.00, lr: 0.000100
[2025-07-31 15:16:42] (step=0005296/epoch=0000) Train Loss: 0.1697, Gradient Norm: 0.0088, Sec/Train Steps: 291.38, lr: 0.000100
[2025-07-31 15:36:07] (step=0005300/epoch=0000) Train Loss: 0.1635, Gradient Norm: 0.0052, Sec/Train Steps: 291.12, lr: 0.000100
[2025-07-31 15:55:33] (step=0005304/epoch=0000) Train Loss: 0.1806, Gradient Norm: 0.0067, Sec/Train Steps: 291.56, lr: 0.000100
[2025-07-31 16:15:04] (step=0005308/epoch=0000) Train Loss: 0.1669, Gradient Norm: 0.0082, Sec/Train Steps: 292.72, lr: 0.000100
[2025-07-31 16:34:32] (step=0005312/epoch=0000) Train Loss: 0.1665, Gradient Norm: 0.0073, Sec/Train Steps: 292.00, lr: 0.000100
[2025-07-31 16:53:57] (step=0005316/epoch=0000) Train Loss: 0.1629, Gradient Norm: 0.0079, Sec/Train Steps: 291.29, lr: 0.000100
[2025-07-31 17:13:24] (step=0005320/epoch=0000) Train Loss: 0.1691, Gradient Norm: 0.0139, Sec/Train Steps: 291.53, lr: 0.000100
[2025-07-31 17:32:47] (step=0005324/epoch=0000) Train Loss: 0.1664, Gradient Norm: 0.0098, Sec/Train Steps: 290.79, lr: 0.000100
[2025-07-31 17:52:12] (step=0005328/epoch=0000) Train Loss: 0.1751, Gradient Norm: 0.0065, Sec/Train Steps: 291.17, lr: 0.000100
[2025-07-31 17:52:15] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005328.pt
[2025-07-31 18:11:42] (step=0005332/epoch=0000) Train Loss: 0.1631, Gradient Norm: 0.0075, Sec/Train Steps: 292.45, lr: 0.000100
[2025-07-31 18:31:09] (step=0005336/epoch=0000) Train Loss: 0.1764, Gradient Norm: 0.0070, Sec/Train Steps: 291.80, lr: 0.000100
[2025-07-31 18:50:39] (step=0005340/epoch=0000) Train Loss: 0.1769, Gradient Norm: 0.0067, Sec/Train Steps: 292.60, lr: 0.000100
[2025-07-31 19:10:10] (step=0005344/epoch=0000) Train Loss: 0.1720, Gradient Norm: 0.0050, Sec/Train Steps: 292.58, lr: 0.000100
[2025-07-31 19:29:36] (step=0005348/epoch=0000) Train Loss: 0.1685, Gradient Norm: 0.0081, Sec/Train Steps: 291.50, lr: 0.000100
[2025-07-31 19:48:58] (step=0005352/epoch=0000) Train Loss: 0.1698, Gradient Norm: 0.0056, Sec/Train Steps: 290.45, lr: 0.000100
[2025-07-31 20:08:25] (step=0005356/epoch=0000) Train Loss: 0.1697, Gradient Norm: 0.0067, Sec/Train Steps: 291.97, lr: 0.000100
[2025-07-31 20:27:52] (step=0005360/epoch=0000) Train Loss: 0.1685, Gradient Norm: 0.0052, Sec/Train Steps: 291.72, lr: 0.000100
[2025-07-31 20:47:18] (step=0005364/epoch=0000) Train Loss: 0.1648, Gradient Norm: 0.0047, Sec/Train Steps: 291.39, lr: 0.000100
[2025-07-31 21:06:44] (step=0005368/epoch=0000) Train Loss: 0.1693, Gradient Norm: 0.0067, Sec/Train Steps: 291.44, lr: 0.000100
[2025-07-31 21:26:12] (step=0005372/epoch=0000) Train Loss: 0.1682, Gradient Norm: 0.0060, Sec/Train Steps: 292.04, lr: 0.000100
[2025-07-31 21:45:41] (step=0005376/epoch=0000) Train Loss: 0.1800, Gradient Norm: 0.0061, Sec/Train Steps: 292.18, lr: 0.000100
[2025-07-31 21:45:44] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005376.pt
[2025-07-31 22:05:14] (step=0005380/epoch=0000) Train Loss: 0.1637, Gradient Norm: 0.0055, Sec/Train Steps: 293.01, lr: 0.000100
[2025-07-31 22:24:38] (step=0005384/epoch=0000) Train Loss: 0.1728, Gradient Norm: 0.0076, Sec/Train Steps: 290.99, lr: 0.000100
[2025-07-31 22:44:02] (step=0005388/epoch=0000) Train Loss: 0.1710, Gradient Norm: 0.0082, Sec/Train Steps: 290.93, lr: 0.000100
[2025-07-31 23:03:31] (step=0005392/epoch=0000) Train Loss: 0.1754, Gradient Norm: 0.0054, Sec/Train Steps: 292.26, lr: 0.000100
[2025-07-31 23:22:57] (step=0005396/epoch=0000) Train Loss: 0.1710, Gradient Norm: 0.0098, Sec/Train Steps: 291.56, lr: 0.000100
[2025-07-31 23:42:24] (step=0005400/epoch=0000) Train Loss: 0.1721, Gradient Norm: 0.0079, Sec/Train Steps: 291.48, lr: 0.000100
[2025-08-01 00:01:52] (step=0005404/epoch=0000) Train Loss: 0.1840, Gradient Norm: 0.0070, Sec/Train Steps: 291.98, lr: 0.000100
[2025-08-01 00:21:19] (step=0005408/epoch=0000) Train Loss: 0.1807, Gradient Norm: 0.0112, Sec/Train Steps: 291.93, lr: 0.000100
[2025-08-01 00:40:45] (step=0005412/epoch=0000) Train Loss: 0.1770, Gradient Norm: 0.0079, Sec/Train Steps: 291.38, lr: 0.000100
[2025-08-01 01:00:11] (step=0005416/epoch=0000) Train Loss: 0.1731, Gradient Norm: 0.0063, Sec/Train Steps: 291.55, lr: 0.000100
[2025-08-01 01:19:35] (step=0005420/epoch=0000) Train Loss: 0.1700, Gradient Norm: 0.0051, Sec/Train Steps: 290.96, lr: 0.000100
[2025-08-01 01:39:03] (step=0005424/epoch=0000) Train Loss: 0.1686, Gradient Norm: 0.0065, Sec/Train Steps: 291.84, lr: 0.000100
[2025-08-01 01:39:06] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005424.pt
[2025-08-01 01:58:33] (step=0005428/epoch=0000) Train Loss: 0.1648, Gradient Norm: 0.0049, Sec/Train Steps: 292.58, lr: 0.000100
[2025-08-01 02:17:59] (step=0005432/epoch=0000) Train Loss: 0.1703, Gradient Norm: 0.0080, Sec/Train Steps: 291.54, lr: 0.000100
[2025-08-01 02:37:26] (step=0005436/epoch=0000) Train Loss: 0.1769, Gradient Norm: 0.0092, Sec/Train Steps: 291.56, lr: 0.000100
[2025-08-01 02:56:54] (step=0005440/epoch=0000) Train Loss: 0.1802, Gradient Norm: 0.0063, Sec/Train Steps: 292.17, lr: 0.000100
[2025-08-01 03:16:27] (step=0005444/epoch=0000) Train Loss: 0.1714, Gradient Norm: 0.0073, Sec/Train Steps: 293.10, lr: 0.000100
[2025-08-01 03:35:51] (step=0005448/epoch=0000) Train Loss: 0.1691, Gradient Norm: 0.0068, Sec/Train Steps: 291.09, lr: 0.000100
[2025-08-01 03:55:17] (step=0005452/epoch=0000) Train Loss: 0.1736, Gradient Norm: 0.0065, Sec/Train Steps: 291.23, lr: 0.000100
[2025-08-01 04:14:49] (step=0005456/epoch=0000) Train Loss: 0.1745, Gradient Norm: 0.0071, Sec/Train Steps: 292.95, lr: 0.000100
[2025-08-01 04:34:12] (step=0005460/epoch=0000) Train Loss: 0.1741, Gradient Norm: 0.0067, Sec/Train Steps: 290.91, lr: 0.000100
[2025-08-01 04:53:40] (step=0005464/epoch=0000) Train Loss: 0.1657, Gradient Norm: 0.0061, Sec/Train Steps: 291.84, lr: 0.000100
[2025-08-01 05:13:10] (step=0005468/epoch=0000) Train Loss: 0.1769, Gradient Norm: 0.0087, Sec/Train Steps: 292.47, lr: 0.000100
[2025-08-01 05:32:36] (step=0005472/epoch=0000) Train Loss: 0.1748, Gradient Norm: 0.0074, Sec/Train Steps: 291.71, lr: 0.000100
[2025-08-01 05:32:40] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005472.pt
[2025-08-01 05:52:09] (step=0005476/epoch=0000) Train Loss: 0.1633, Gradient Norm: 0.0062, Sec/Train Steps: 293.15, lr: 0.000100
[2025-08-01 06:11:31] (step=0005480/epoch=0000) Train Loss: 0.1637, Gradient Norm: 0.0069, Sec/Train Steps: 290.53, lr: 0.000100
[2025-08-01 06:30:59] (step=0005484/epoch=0000) Train Loss: 0.1692, Gradient Norm: 0.0062, Sec/Train Steps: 291.93, lr: 0.000100
[2025-08-01 06:50:27] (step=0005488/epoch=0000) Train Loss: 0.1747, Gradient Norm: 0.0044, Sec/Train Steps: 291.80, lr: 0.000100
[2025-08-01 07:09:52] (step=0005492/epoch=0000) Train Loss: 0.1699, Gradient Norm: 0.0076, Sec/Train Steps: 291.26, lr: 0.000100
[2025-08-01 07:29:21] (step=0005496/epoch=0000) Train Loss: 0.1704, Gradient Norm: 0.0068, Sec/Train Steps: 292.41, lr: 0.000100
[2025-08-01 07:48:48] (step=0005500/epoch=0000) Train Loss: 0.1774, Gradient Norm: 0.0082, Sec/Train Steps: 291.73, lr: 0.000100
[2025-08-01 08:08:13] (step=0005504/epoch=0000) Train Loss: 0.1682, Gradient Norm: 0.0046, Sec/Train Steps: 291.15, lr: 0.000100
[2025-08-01 08:27:43] (step=0005508/epoch=0000) Train Loss: 0.1705, Gradient Norm: 0.0078, Sec/Train Steps: 292.47, lr: 0.000100
[2025-08-01 08:47:15] (step=0005512/epoch=0000) Train Loss: 0.1655, Gradient Norm: 0.0048, Sec/Train Steps: 293.10, lr: 0.000100
[2025-08-01 09:06:45] (step=0005516/epoch=0000) Train Loss: 0.1706, Gradient Norm: 0.0066, Sec/Train Steps: 292.56, lr: 0.000100
[2025-08-01 09:26:11] (step=0005520/epoch=0000) Train Loss: 0.1656, Gradient Norm: 0.0072, Sec/Train Steps: 291.32, lr: 0.000100
[2025-08-01 09:26:14] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005520.pt
[2025-08-01 09:45:43] (step=0005524/epoch=0000) Train Loss: 0.1677, Gradient Norm: 0.0057, Sec/Train Steps: 292.97, lr: 0.000100
[2025-08-01 10:05:10] (step=0005528/epoch=0000) Train Loss: 0.1754, Gradient Norm: 0.0088, Sec/Train Steps: 291.59, lr: 0.000100
[2025-08-01 10:24:33] (step=0005532/epoch=0000) Train Loss: 0.1699, Gradient Norm: 0.0054, Sec/Train Steps: 290.81, lr: 0.000100
[2025-08-01 10:43:58] (step=0005536/epoch=0000) Train Loss: 0.1749, Gradient Norm: 0.0092, Sec/Train Steps: 291.22, lr: 0.000100
[2025-08-01 11:03:23] (step=0005540/epoch=0000) Train Loss: 0.1679, Gradient Norm: 0.0081, Sec/Train Steps: 291.38, lr: 0.000100
[2025-08-01 11:22:54] (step=0005544/epoch=0000) Train Loss: 0.1738, Gradient Norm: 0.0058, Sec/Train Steps: 292.76, lr: 0.000100
[2025-08-01 11:42:19] (step=0005548/epoch=0000) Train Loss: 0.1765, Gradient Norm: 0.0097, Sec/Train Steps: 291.25, lr: 0.000100
[2025-08-01 12:01:45] (step=0005552/epoch=0000) Train Loss: 0.1785, Gradient Norm: 0.0091, Sec/Train Steps: 291.49, lr: 0.000100
[2025-08-01 12:21:15] (step=0005556/epoch=0000) Train Loss: 0.1711, Gradient Norm: 0.0049, Sec/Train Steps: 292.17, lr: 0.000100
[2025-08-01 12:40:42] (step=0005560/epoch=0000) Train Loss: 0.1781, Gradient Norm: 0.0114, Sec/Train Steps: 291.80, lr: 0.000100
[2025-08-01 13:00:10] (step=0005564/epoch=0000) Train Loss: 0.1778, Gradient Norm: 0.0079, Sec/Train Steps: 292.06, lr: 0.000100
[2025-08-01 13:19:43] (step=0005568/epoch=0000) Train Loss: 0.1748, Gradient Norm: 0.0075, Sec/Train Steps: 293.31, lr: 0.000100
[2025-08-01 13:19:47] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005568.pt
[2025-08-01 13:39:12] (step=0005572/epoch=0000) Train Loss: 0.1746, Gradient Norm: 0.0080, Sec/Train Steps: 292.11, lr: 0.000100
[2025-08-01 13:58:38] (step=0005576/epoch=0000) Train Loss: 0.1760, Gradient Norm: 0.0064, Sec/Train Steps: 291.41, lr: 0.000100
[2025-08-01 14:18:06] (step=0005580/epoch=0000) Train Loss: 0.1656, Gradient Norm: 0.0092, Sec/Train Steps: 291.81, lr: 0.000100
[2025-08-01 14:37:32] (step=0005584/epoch=0000) Train Loss: 0.1740, Gradient Norm: 0.0084, Sec/Train Steps: 291.65, lr: 0.000100
[2025-08-01 14:57:01] (step=0005588/epoch=0000) Train Loss: 0.1744, Gradient Norm: 0.0093, Sec/Train Steps: 292.09, lr: 0.000100
[2025-08-01 15:16:28] (step=0005592/epoch=0000) Train Loss: 0.1726, Gradient Norm: 0.0058, Sec/Train Steps: 291.69, lr: 0.000100
[2025-08-01 15:35:52] (step=0005596/epoch=0000) Train Loss: 0.1717, Gradient Norm: 0.0067, Sec/Train Steps: 290.99, lr: 0.000100
[2025-08-01 15:55:20] (step=0005600/epoch=0000) Train Loss: 0.1670, Gradient Norm: 0.0058, Sec/Train Steps: 291.97, lr: 0.000100
[2025-08-01 16:14:48] (step=0005604/epoch=0000) Train Loss: 0.1737, Gradient Norm: 0.0088, Sec/Train Steps: 292.01, lr: 0.000100
[2025-08-01 16:34:17] (step=0005608/epoch=0000) Train Loss: 0.1715, Gradient Norm: 0.0126, Sec/Train Steps: 292.13, lr: 0.000100
[2025-08-01 16:53:44] (step=0005612/epoch=0000) Train Loss: 0.1748, Gradient Norm: 0.0073, Sec/Train Steps: 291.90, lr: 0.000100
[2025-08-01 17:13:11] (step=0005616/epoch=0000) Train Loss: 0.1632, Gradient Norm: 0.0078, Sec/Train Steps: 291.47, lr: 0.000100
[2025-08-01 17:13:14] Saved checkpoint to ./lora_log/012-Wan2.1-1.3B-F81/checkpoints/0005616.pt
[2025-08-01 17:32:40] (step=0005620/epoch=0000) Train Loss: 0.1689, Gradient Norm: 0.0063, Sec/Train Steps: 292.27, lr: 0.000100
[2025-08-01 17:52:04] (step=0005624/epoch=0000) Train Loss: 0.1617, Gradient Norm: 0.0041, Sec/Train Steps: 290.96, lr: 0.000100
[2025-08-01 18:11:29] (step=0005628/epoch=0000) Train Loss: 0.1725, Gradient Norm: 0.0047, Sec/Train Steps: 291.34, lr: 0.000100
[2025-08-01 18:30:55] (step=0005632/epoch=0000) Train Loss: 0.1762, Gradient Norm: 0.0085, Sec/Train Steps: 291.47, lr: 0.000100
[2025-08-01 18:50:22] (step=0005636/epoch=0000) Train Loss: 0.1708, Gradient Norm: 0.0068, Sec/Train Steps: 291.67, lr: 0.000100
