[refactor] Apply the #95 scheme in deepspeed &add single file copy feature#128
[refactor] Apply the #95 scheme in deepspeed &add single file copy feature#128zheliuyu merged 2 commits intoAscend:mainfrom
Conversation
CLA Signature Passzheliuyu, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
zheliuyu
changed the title
CLA Signature Passzheliuyu, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
| <p class="card-desc">分布式训练优化库，V0.10.1 版本起支持昇腾。</p> | ||
| <div class="card-footer"><a href="https://github.com/microsoft/DeepSpeed">官方链接</a><span class="split">|</span><a href="sources/deepspeed/install.html">安装指南</a><span class="split">|</span> | ||
| <p class="card-desc">DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. </p> | ||
| <div class="card-footer"><a href="https://github.com/deepspeedai/DeepSpeed">官方链接</a><span class="split">|</span><a href="sources/deepspeed/quick_start.html">安装指南</a><span class="split">|</span> |
There was a problem hiding this comment.
The remote URL for deepspeed has changed and has been updated here.
| rm -rf "$$dst"; \ | ||
| if [ -f "$$src" ]; then \ | ||
| mkdir -p "$$(dirname "$$dst")"; \ | ||
| echo "Copying $$src to $$dst"; \ | ||
| cp "$$src" "$$dst" || echo " [WARN] Source file missing or copy failed: $$src"; \ | ||
| elif [ -d "$$src" ]; then \ | ||
| mkdir -p "$$dst"; \ | ||
| echo "Copying $$src to $$dst"; \ | ||
| cp -r "$$src"/* "$$dst"/ 2>/dev/null || echo " [WARN] Source directory does not exist or is empty: $$src"; \ | ||
| else \ | ||
| echo " [WARN] Source does not exist: $$src"; \ | ||
| fi; \ |
There was a problem hiding this comment.
In addition to updating the deepspeed documentation, this adds a single file copy feature.
CLA Signature Passzheliuyu, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Passzheliuyu, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
CLA Signature Passzheliuyu, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
| # _repos: submodule working trees duplicate content already copied to sources/_generated; | ||
| # indexing both roughly doubles RSS and OOM-kills low-memory cgroup builds (e.g. 2Gi). | ||
| exclude_patterns = ['_build', '_repos', 'Thumbs.db', '.DS_Store', '.venv', 'README.md'] |
|
The official DeepSpeed documentation needs to be updated. However, the goal of this PR is to enable DeepSpeed to be updated via git submodule. Let's merge this PR first to unblock the subsequent tasks. |
2 participants
Motivation
Update deepspeed doc. Apply the #95 scheme
Add single file copy feature to Makefile to reduce memory and disk usage.