Search

huggingface-pytorch-tpuv3x8

train steps
114
batch-size
128
samples/sec
Empty
train-time(sec)
66.8629
x faster
Empty
python xla_spawn.py --num_cores 8 \ language-modeling/run_mlm.py \ --model_name_or_path roberta-base\ --dataset_name wikitext\ --dataset_config_name wikitext-2-raw-v1\ --do_train\ --do_eval\ --output_dir ../test-result\ --per_device_train_batch_size=16
Shell
[INFO|trainer.py:837] 2021-02-20 13:20:05,089 >> ***** Running training ***** [INFO|trainer.py:838] 2021-02-20 13:20:05,089 >> Num examples = 4798 [INFO|trainer.py:839] 2021-02-20 13:20:05,089 >> Num Epochs = 3 [INFO|trainer.py:840] 2021-02-20 13:20:05,089 >> Instantaneous batch size per device = 16 [INFO|trainer.py:841] 2021-02-20 13:20:05,089 >> Total train batch size (w. parallel, distributed & accumulation) = 128 [INFO|trainer.py:842] 2021-02-20 13:20:05,089 >> Gradient Accumulation steps = 1 [INFO|trainer.py:843] 2021-02-20 13:20:05,089 >> Total optimization steps = 114 {'train_runtime': 66.8629, 'train_samples_per_second': 1.705, 'epoch': 3.0} 100%|#######################################################################################################################| 114/114 [01:06<00:00, 1.71it/s] [INFO|trainer.py:1387] 2021-02-20 13:21:12,151 >> Saving model checkpoint to ../test-result [INFO|configuration_utils.py:304] 2021-02-20 13:21:12,205 >> Configuration saved in ../test-result/config.json [INFO|modeling_utils.py:817] 2021-02-20 13:21:17,246 >> Model weights saved in ../test-result/pytorch_model.bin 02/20/2021 13:21:17 - INFO - run_mlm - ***** Train results ***** 02/20/2021 13:21:17 - INFO - run_mlm - epoch = 3.0 02/20/2021 13:21:17 - INFO - run_mlm - train_runtime = 66.8629 02/20/2021 13:21:17 - INFO - run_mlm - train_samples_per_second = 1.705 02/20/2021 13:21:17 - INFO - run_mlm - *** Evaluate *** [INFO|trainer.py:1600] 2021-02-20 13:21:17,393 >> ***** Running Evaluation ***** [INFO|trainer.py:1601] 2021-02-20 13:21:17,393 >> Num examples = 496 [INFO|trainer.py:1602] 2021-02-20 13:21:17,394 >> Batch size = 8 100%|###########################################################################################################################| 8/8 [00:01<00:00, 5.74it/s] 02/20/2021 13:21:19 - INFO - run_mlm - ***** Eval results ***** 02/20/2021 13:21:19 - INFO - run_mlm - perplexity = 3.5044896037251934
Shell