By default checkpoints are periodically written to the -out_dir. Your multinode training will work, but most likely crawl. In particular, if you don't have Infiniband then also prepend NCCL_IB_DISABLE=1 to the above launches. It is a good idea to benchmark your interconnect (e.g. Run on the first (master) node with example IP 123.456.123.456: This still runs in about ~3 minutes, but gets us a loss of only 1.88 and therefore also worse samples, but it's still good fun: Because our network is so small we also ease down on regularization ( -dropout=0.0). We'll also use a much smaller Transformer (4 layers, 4 heads, 128 embedding size), and decrease the number of iterations to 2000 (and correspondingly usually decay the learning rate to around max_iters with -lr_decay_iters). Then when we evaluate we get a bit more noisy but faster estimate ( -eval_iters=20, down from 200), our context size is only 64 characters instead of 256, and the batch size only 12 examples per iteration, not 64. Here, since we are running on CPU instead of GPU we must set both -device=cpu and also turn off PyTorch 2.0 compile with -compile=False. $ python train.py config/train_shakespeare_char.py -device=cpu -compile=False -eval_iters=20 -log_interval=1 -block_size=64 -batch_size=12 -n_layer=4 -n_head=4 -n_embd=128 -max_iters=2000 -lr_decay_iters=2000 -dropout=0.0 But even without it, a simple train run could look as follows: I recommend getting the bleeding edge PyTorch nightly ( select it here when installing) as it is currently quite likely to make your code more efficient. No worries, we can still train a GPT but we want to dial things down a notch. I only have a macbook (or other cheap computer). Better results are quite likely obtainable by instead finetuning a pretrained GPT-2 model on this dataset (see finetuning section later). Not bad for a character-level model after 3 minutes of training on a GPU. To end his power, the day of thrust for a common men If you have done evils of all disposition And what have you tyrannous shall do this?
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |