Merge pull request #3 from Alpha-VLLM/main

compare diff

Author: kriskrisliu

README.md

@@ -14,7 +14,7 @@
 ## Features
 * **💡Support More Datasets and Tasks**
   - 🎯 Pre-training with [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) and [StarCoder](https://github.com/bigcode-project/starcoder).
-  - 📚 Single-modal fine-tuning with [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), [ShareGPT](https://github.com/domeccleston/sharegpt), [LIMA](https://arxiv.org/pdf/2305.11206.pdf), [WizardLM](https://github.com/nlpxucan/WizardLM), [UltraChat](https://github.com/thunlp/UltraChat) and [MOSS](https://github.com/OpenLMLab/MOSS).
+  - 📚 Single-modal fine-tuning with [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), [ShareGPT](https://github.com/domeccleston/sharegpt), [LIMA](https://arxiv.org/pdf/2305.11206.pdf), [WizardLM](https://github.com/nlpxucan/WizardLM), [Flacuna](https://github.com/declare-lab/flacuna), [Platypus](https://github.com/arielnlee/Platypus), [UltraChat](https://github.com/thunlp/UltraChat) and [MOSS](https://github.com/OpenLMLab/MOSS).
   - 🌈 Multi-modal fine-tuning with image-text pairs ([LAION](https://laion.ai/blog/laion-5b/), [COYO](https://github.com/kakaobrain/coyo-dataset) and more), interleaved image-text data ([MMC4](https://github.com/allenai/mmc4) and [OBELISC](https://github.com/huggingface/OBELISC)) and visual instruction data ([LLaVA](https://github.com/haotian-liu/LLaVA), [Shrika](https://github.com/shikras/shikra), [Bard](https://bard.google.com/))
   - 🔧 LLM for API Control ([GPT4Tools](https://github.com/StevenGrove/GPT4Tools) and [Gorilla](https://github.com/ShishirPatil/gorilla)).
 * **⚡Efficient Optimization and Deployment**
@@ -112,6 +112,7 @@ If you find our code and paper useful, please kindly cite:
 + [@microsoft](https://github.com/microsoft) for [DeepSpeed](https://github.com/microsoft/DeepSpeed)
 + [@declare-lab](https://github.com/declare-lab) for [flacuna](https://github.com/declare-lab/flacuna)
 + [@nlpxucan](https://github.com/nlpxucan) for [WizardLM](https://github.com/nlpxucan/WizardLM)
++ [@arielnlee](https://github.com/arielnlee) for [Platypus](https://github.com/arielnlee/Platypus)
 + [@Google](https://github.com/google) for [Bard](https://bard.google.com/)
 </details>
 

accessory/configs/data/finetune/sg/platypus.yaml

@@ -0,0 +1,2 @@
+META:
+  - ['../data/Platypus_alpaca_format.json', 'text']

accessory/demos/single_turn_mm.py

@@ -14,7 +14,7 @@
 from util import misc
 from fairscale.nn.model_parallel import initialize as fs_init
 
-from data.alpaca import transform_train, format_prompt
+from data.alpaca import transform_val, format_prompt
 from util.tensor_parallel import load_tensor_parallel_model
 from util.quant import quantize
 
@@ -86,7 +86,7 @@ def generate(
 ):
     if img_path is not None:
         image = Image.open(img_path).convert('RGB')
-        image = transform_train(image).unsqueeze(0)
+        image = transform_val(image).unsqueeze(0)
     else:
         image = None
 

accessory/exps/finetune/sg/platypus.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+pretrained_path=$1
+pretrained_type=meta_ori
+llama_config="$2"
+tokenizer_path="$3"
+data_config=configs/data/finetune/sg/platypus.yaml
+
+data_parallel=sdp
+model_parallel=1
+
+exp_name=finetune/sg/platypus
+echo "exp name: $exp_name"
+mkdir -p output/"$exp_name"
+
+torchrun --master_port=1112 --nproc_per_node=6 main_finetune.py \
+--output_dir output/"$exp_name" --epochs 4 --warmup_epochs 1 \
+--batch_size 4 --accum_iter 2 --num_workers 4 \
+--max_words 512 \
+--lr 0.00003 --min_lr 0.000005 --clip_grad 2 --weight_decay 0.02 \
+--data_parallel "$data_parallel" --model_parallel_size "$model_parallel" --checkpointing \
+--llama_type llama --llama_config "$llama_config" --tokenizer_path "$tokenizer_path" \
+--no_visual \
+--pretrained_path "$pretrained_path" --pretrained_type="$pretrained_type" \
+--data_config $data_config \
+2>&1 | tee -a output/"$exp_name"/output.log
+
+echo "exp name: $exp_name"

accessory/tools/download.py

@@ -20,6 +20,7 @@ def get_args_parser():
     parser.add_argument('--input_type', default='sg', choices=['sg', 'mm'])
     parser.add_argument('--model_size', default='7B', choices=['7B', '13B', '70B'])
     parser.add_argument('--down_config', action="store_true" ,help='download config')
+    parser.add_argument('--down_diff', action="store_true" ,help='download delta weights')
     return parser
 
 if __name__ == '__main__':
@@ -41,7 +42,10 @@ def get_args_parser():
     max_num = num_files_map[args.model_size]
 
     for num in range(max_num):
-        file_name = f"consolidated.{num:02d}-of-{max_num:02d}.model-diff.pth"
+        if args.down_diff:
+            file_name = f"consolidated.{num:02d}-of-{max_num:02d}.model-diff.pth"
+        else:
+            file_name = f"consolidated.{num:02d}-of-{max_num:02d}.model.pth"
         download_file(repo_id, subfolder, file_name, args.output_path)
 
     print(f"{args.model_name} model files downloaded successfully to {args.output_path}")