WSL 上通过 nvidia docker 镜像做 VITS 项目训练尝试

环境准备： win11, wsl 2.3.26.0,Docker version 26.1.3
VITS 库：https://github.com/jaywalnut310
NVidia docker 镜像：11.1.1-base-ubuntu16.04

采用 docker 镜像方式运行的原因，笔记本电脑上有一个 mx150 显卡，想尝试以下运行机器学习。由于使用 gpu 做训练，一些依赖的软件库未找到windows 版本，需要自己下载源代码编译，故使用 nvidia 提供的 镜像，免受折腾之苦。

目的：nvidia container 运行 vits 训练：

1.wsl 下执行如下命令，启动docker 容器：

docker run --gpus all -it --sysctl kernel.shmmax=4294967296 --shm-size=4gb --sysctl kernel.shmall=1048576 --rm -v /mnt/d/dev/git/learn/tts/VITS:/VITS nvidia/cuda:11.1.1-base-ubuntu16.04 /bin/bash

说明：/mnt/d/dev/git/learn/tts/VITS 为wsl 目录下 下载的 VITS 源代码
参考：
sudo sysctl -w kernel.shmmax=4294967296 # 将共享内存最大值设置为 4GB
sudo sysctl -w kernel.shmall=1048576 # 将共享内存段总数设置为 1GB 页面
--shm-size=4gb 将 docker 的共享内存改为 4 gb, 默认为 64 M
docker inspect <容器ID> | grep ShmSize

2.在容器中执行：

安装 Python3.8.10
ubuntu 上安装python 3.8.12
使用 apt 方式可能会提示找不到 包，可以通过手动方式下载安装：
查找 python 发布版本：https://www.python.org/downloads/release/python-3812/

apt-get update
apt install curl -y
curl -O https://www.python.org/ftp/python/3.8.12/Python-3.8.12.tar.xz
apt-get install -y xz-utils //安装 xz 解压工具
tar Jxf Python-3.8.12.tar.xz

#安装编译所需的库
apt-get install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev \
libnss3-dev libssl-dev libreadline-dev libffi-dev wget curl llvm \
libsqlite3-dev tk-dev libmpdec-dev libbz2-dev libexpat1-dev liblzma-dev cmake git

# 安装 CUDA 开发工具, nvidia base 镜像，不包括 nvcc 等开发工具，需要手动安装，安装好之后，可执行命令：nvcc --version 验证

apt-get install -y cuda-command-line-tools-11-1

生成 makefile 及配置文件

cd Python-3.8.12

./configure prefix=/usr/local/python

make

make install

ln -s /usr/local/python/bin/python3.8 /usr/bin/python

ln -s /usr/local/python/bin/pip3 /usr/bin/pip

cd /VITS

设置 pip 国内镜像：

新建文件
mkdir -p ~/.pip
cat <<EOL > ~/.pip/pip.conf
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
EOL

更新 requirements.txt 内容，然后执行 pip install -r requirements.txt

Cython==3.0.7
librosa==0.10.0
matplotlib==3.3.1
numpy==1.21.0
phonemizer==2.2.1
scipy==1.5.2
tensorboard==2.3.0
torch==2.4.1
torchvision==0.19.1
torchaudio==2.4.1
Unidecode==1.3.7
protobuf==3.20.0
urllib3==1.26.15

链接训练文件
ln -s LJSpeech-1.1/wavs/ DUMMY1
训练：
将 train.py 的os.environ['MASTER_PORT'] = '80000' 改为 os.environ['MASTER_PORT'] = '8001'
python train.py -c configs/ljs_base.json -m ljs_base

以下为错误排查

======================================================================================================================

提示如下错误，安装：pip install protobuf==3.20.0

TypeError: Descriptors cannot be created directly.

If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.

If you cannot immediately regenerate your protos, some other possible workarounds are:

1. Downgrade the protobuf package to 3.20.x or lower.

降级 urllib3：pip install urllib3==1.26.15

ImportError: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2g 1 Mar 2016'. See: https://github.com/urllib3/urllib3/issues/2168

2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

monotonic_align 代码由 cyphon 所写，需要安装 Cython 代码所需的工具和库，并编译 monotonic_align

apt install -y cython3

cd monotonic_align

python setup.py build_ext --inplace

未编译则会出现如下错误：

Traceback (most recent call last):

File "train.py", line 23, in <module>

from models import (

File "/VITS/models.py", line 10, in <module>

import monotonic_align

File "/VITS/monotonic_align/__init__.py", line 3, in <module>

from .monotonic_align.core import maximum_path_c

ModuleNotFoundError: No module named 'monotonic_align.monotonic_align.core'

==========================================================================

出现错误：

-- Process 0 terminated with the following error:

Traceback (most recent call last):

File "/usr/local/python/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap

fn(i, *args)

File "/VITS/train.py", line 99, in run

net_g = DDP(net_g, device_ids=[rank])

File "/usr/local/python/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 331, in __init__

self._distributed_broadcast_coalesced(

File "/usr/local/python/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 549, in _distributed_broadcast_coalesced

dist._broadcast_coalesced(self.process_group, tensors, buffer_size)

RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:518, unhandled system error, NCCL version 2.4.8

做如下操作：

安装和构建 nccl-tests，nccl 和 cuda 版本关系，查看下载链接：https://developer.nvidia.com/nccl/nccl-legacy-downloads

#git clone https://github.com/NVIDIA/nccl.git # master

为了 与容器的 cuda 版本匹配，本次选择 分支 v2.16

git clone --branch v2.16 https://github.com/NVIDIA/nccl.git

cd nccl

构建 NCCL

make -j src.build #该方法默认会编译全部 架构，可以用如下命令减少时间：

make -j src.build NVCC_GENCODE="-gencode=arch=compute_70,code=sm_70"

Install

To install NCCL on the system, create a package then install it as root.

Debian/Ubuntu :

$ # Install tools to create debian packages

$ sudo apt install build-essential devscripts debhelper fakeroot

$ # Build NCCL deb package

$ make pkg.debian.build

$ ls build/pkg/deb/

或

make pkg.txz.build

ls build/pkg/txz/

make install build/pkg/txz/nccl_2.16.2-1+cuda11.1_x86_64.txz

Tests for NCCL

git clone https://github.com/NVIDIA/nccl-tests.git

cd nccl-tests

make

export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH

./build/all_reduce_perf -b 8 -e 256M -f 2 -g 1

执行结果都正常，出现以上错误是因为

可以运行以下命令升级 torch,cuda

pip install --upgrade torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu101

升级之前版本信息：

PyTorch 版本：1.6.0+cu92

CUDA 版本：9.2

NCCL 版本：2.4.8

torchvision: 0.7.0

torchaudio：0.6.0

升级之后版本信息：

PyTorch 版本：2.4.1+cu121

CUDA 版本：12.1

NCCL 版本：2.20.5

torchvision: 0.19.1+cu121

torchaudio：2.4.1+cu121

==========================================================================

默认情况下 cuda 安装在 目录 /usr/local/cuda 下

出现如下错误，修改 data_utils.py，调整代码，增加 0 值判断，如果为 0， 跳过：

or i in range(len(self.buckets)):

bucket = self.buckets[i]

len_bucket = len(bucket)

if(len_bucket == 0):

continue

-- Process 0 terminated with the following error:

Traceback (most recent call last):

File "/usr/local/python/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 76, in _wrap

fn(i, *args)

File "/VITS/train.py", line 118, in run

train_and_evaluate(rank, epoch, hps, [net_g, net_d], [optim_g, optim_d], [scheduler_g, scheduler_d], scaler, [train_loader, eval_loader], logger, [writer, writer_eval])

File "/VITS/train.py", line 138, in train_and_evaluate

for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths) in enumerate(train_loader):

File "/usr/local/python/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 440, in __iter__

return self._get_iterator()

File "/usr/local/python/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 388, in _get_iterator

return _MultiProcessingDataLoaderIter(self)

File "/usr/local/python/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in __init__

super().__init__(loader)

File "/usr/local/python/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 600, in __init__

self._sampler_iter = iter(self._index_sampler)

File "/VITS/data_utils.py", line 358, in __iter__

ids_bucket = ids_bucket + ids_bucket * (rem // len_bucket) + ids_bucket[:(rem % len_bucket)]

ZeroDivisionError: integer division or modulo by zero

=================================================================================

train.py 代码调整：

os.environ['MASTER_PORT'] = '8001'

#utils.check_git_hash(hps.model_dir)

train_loader = DataLoader(train_dataset, num_workers=1, shuffle=False, pin_memory=False,

collate_fn=collate_fn, batch_sampler=train_sampler,prefetch_factor=1)

if rank == 0:

eval_dataset = TextAudioLoader(hps.data.validation_files, hps.data)

eval_loader = DataLoader(eval_dataset, num_workers=1, shuffle=False,

batch_size=hps.train.batch_size, pin_memory=False,

drop_last=False, collate_fn=collate_fn)

utils.py 调整：

data_utils.py 调整:

DistributedBucketSampler：

for i in range(len(self.buckets)):

bucket = self.buckets[i]

len_bucket = len(bucket)

if(len_bucket == 0):

continue

ljs_base.json:

train -> batch_size (64 -> 1)

train ->epochs (20000 ->2)

mel_processing.py：

由于 librosa 版本问题，librosa_mel_fn 由位置参数改为关键字参数传递，所有关于 librosa_mel_fn 的调用，均做如下调整：

librosa_mel_fn(sr=sampling_rate, n_fft=n_fft, n_mels=num_mels, fmin=fmin, fmax=fmax)

mel_spectrogram_torch：

spec = torch.stft(y, n_fft, hop_length=hop_size, win_length=win_size, window=hann_window[wnsize_dtype_device],

center=center, pad_mode='reflect', normalized=False, onesided=True,return_complex=False)