Skip to content

Fix Fused Attention Error #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix Fused Attention Error #15

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions pretrain/installers/v3-megatron-sakura/scripts/environment.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@

export PRETRAIN_CUDA_VERSION_MAJOR=12
export PRETRAIN_CUDA_VERSION_MINOR=1
export PRETRAIN_CUDA_VERSION=${PRETRAIN_CUDA_VERSION_MAJOR}.${PRETRAIN_CUDA_VERSION_MINOR}
export PRETRAIN_CUDA_VERSION_PATCH=1
export PRETRAIN_CUDA_VERSION=${PRETRAIN_CUDA_VERSION_MAJOR}.${PRETRAIN_CUDA_VERSION_MINOR}.${PRETRAIN_CUDA_VERSION_PATCH}
export PRETRAIN_CUDA_VERSION_SHORT=${PRETRAIN_CUDA_VERSION_MAJOR}${PRETRAIN_CUDA_VERSION_MINOR}
export PRETRAIN_CUDNN_VERSION=8.9.4
export PRETRAIN_CUDNN_VERSION=8.9.7
export PRETRAIN_HPCX_VERSION=2.17.1
export PRETRAIN_NCCL_VERSION=2.20.5
export PRETRAIN_NCCL_VERSION=2.18.3

export PRETRAIN_PYTHON_VERSION=3.10.14
export PRETRAIN_TORCH_VERSION=2.3.1
Expand All @@ -17,7 +18,8 @@ export PRETRAIN_TRANSFORMER_ENGINE_VERSION=1.4
export PRETRAIN_MEGATRON_TAG=nii-geniac
export PRETRAIN_TOKENIZER_TAG=Release-ver3.0b1

module load cuda/${PRETRAIN_CUDA_VERSION}
module load /data/cudnn-tmp-install/modulefiles/${PRETRAIN_CUDNN_VERSION}
module load hpcx/${PRETRAIN_HPCX_VERSION}-gcc-cuda${PRETRAIN_CUDA_VERSION_MAJOR}/hpcx
module load nccl/${PRETRAIN_NCCL_VERSION}
export MODULEPATH=/data/modules:${MODULEPATH}
module load cuda-${PRETRAIN_CUDA_VERSION}
module load cudnn-${PRETRAIN_CUDNN_VERSION}
module load hpcx-${PRETRAIN_HPCX_VERSION}
module load nccl-${PRETRAIN_NCCL_VERSION}