sbjkx
/
mindspore
forked from OSSInnovation/mindspore
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Refactor docker image framework 

Signed-off-by: leonwanghui <wanghui71leon@gmail.com>
This commit is contained in:
leonwanghui 2020-04-16 16:10:46 +08:00
parent 642e6e7e63
commit c95a04f215
7 changed files with 160 additions and 73 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@ -70,7 +70,7 @@ MindSpore offers build options across multiple backends:
| GPU CUDA 10.1 | Ubuntu-x86 | ✔️ | | GPU CUDA 10.1 | Ubuntu-x86 | ✔️ |
| CPU | Ubuntu-x86 | ✔️ | | CPU | Ubuntu-x86 | ✔️ |
For installation using `pip`, take `Ubuntu-x86` and `CPU` build version as an example: For installation using `pip`, take `CPU` and `Ubuntu-x86` build version as an example:
1. Download whl from [MindSpore download page](https://www.mindspore.cn/versions/en), and install the package. 1. Download whl from [MindSpore download page](https://www.mindspore.cn/versions/en), and install the package.
@ -97,9 +97,11 @@ currently the containerized build options are supported as follows:
| :---------------- | :---------------------- | :-- | :---------- | | :---------------- | :---------------------- | :-- | :---------- |
| CPU | `mindspore/mindspore-cpu` | `0.1.0-alpha` | Production environment with pre-installed MindSpore `0.1.0-alpha` CPU release. | | CPU | `mindspore/mindspore-cpu` | `0.1.0-alpha` | Production environment with pre-installed MindSpore `0.1.0-alpha` CPU release. |
| | | `devel` | Development environment provided to build MindSpore (with `CPU` backend) from the source, refer to https://www.mindspore.cn/install/en for installation details. | | | | `devel` | Development environment provided to build MindSpore (with `CPU` backend) from the source, refer to https://www.mindspore.cn/install/en for installation details. |
| | | `runtime` | Runtime environment provided to install MindSpore binary package with `CPU` backend. |
| GPU | `mindspore/mindspore-gpu` | `0.1.0-alpha` | Production environment with pre-installed MindSpore `0.1.0-alpha` GPU release. | | GPU | `mindspore/mindspore-gpu` | `0.1.0-alpha` | Production environment with pre-installed MindSpore `0.1.0-alpha` GPU release. |
| | | `devel` | Development environment provided to build MindSpore (with `GPU CUDA10.1` backend) from the source, refer to https://www.mindspore.cn/install/en for installation details. | | | | `devel` | Development environment provided to build MindSpore (with `GPU CUDA10.1` backend) from the source, refer to https://www.mindspore.cn/install/en for installation details. |
| Ascend | <center>—</center> | <center>—</center> | Coming soon. | | | | `runtime` | Runtime environment provided to install MindSpore binary package with `GPU` backend. |
| Ascend | <center>&mdash;</center> | <center>&mdash;</center> | Coming soon. |
* CPU * CPU
@ -111,7 +113,7 @@ currently the containerized build options are supported as follows:
* GPU * GPU
For `GPU` backend, please make sure the `nvidia-container-toolkit` has been installed in advance, here are some install guidelines for Ubuntu users: For `GPU` backend, please make sure the `nvidia-container-toolkit` has been installed in advance, here are some install guidelines for `Ubuntu` users:
``` ```
DISTRIBUTION=$(. /etc/os-release; echo $ID$VERSION_ID) DISTRIBUTION=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add -
@ -153,7 +155,7 @@ currently the containerized build options are supported as follows:
[ 2. 2. 2. 2.]]] [ 2. 2. 2. 2.]]]
``` ```
If anyone wants to learn more about the build process of MindSpore docker images, If you want to learn more about the building process of MindSpore docker images,
please check out `docker` folder for the details. please check out `docker` folder for the details.
## Quickstart ## Quickstart

18
docker/mindspore-cpu/0.1.0-alpha/Dockerfile
View File

@ -4,8 +4,7 @@ MAINTAINER leonwanghui <leon.wanghui@huawei.com>
# Set env # Set env
ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5 ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5
ENV CMAKE_ROOT_PATH /usr/local/cmake-3.14.1 ENV PATH /usr/local/bin:$PATH
ENV PATH ${PYTHON_ROOT_PATH}/bin:${CMAKE_ROOT_PATH}/bin:/usr/local/bin:$PATH
# Install base tools # Install base tools
RUN apt update \ RUN apt update \
@ -64,20 +63,5 @@ RUN mkdir -pv /root/.pip \
&& echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \ && echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \
&& echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf && echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf
# Install pip package
RUN pip install --no-cache-dir \
numpy \
wheel \
nose \
pytest \
pytest-xdist
# Install cmake (v3.14.1)
RUN cd /tmp \
&& wget https://github.com/Kitware/CMake/releases/download/v3.14.1/cmake-3.14.1-Linux-x86_64.sh \
&& mkdir -p ${CMAKE_ROOT_PATH} \
&& bash ./cmake-3.14.1-Linux-x86_64.sh --prefix=${CMAKE_ROOT_PATH} --exclude-subdir --skip-license \
&& rm -f /tmp/cmake-3.14.1-Linux-x86_64.sh
# Install MindSpore cpu whl package # Install MindSpore cpu whl package
RUN pip install --no-cache-dir https://ms-release.obs.cn-north-4.myhuaweicloud.com/0.1.0-alpha/MindSpore/cpu/ubuntu-x86/mindspore-0.1.0-cp37-cp37m-linux_x86_64.whl RUN pip install --no-cache-dir https://ms-release.obs.cn-north-4.myhuaweicloud.com/0.1.0-alpha/MindSpore/cpu/ubuntu-x86/mindspore-0.1.0-cp37-cp37m-linux_x86_64.whl

11
docker/mindspore-cpu/devel/Dockerfile
View File

@ -62,15 +62,8 @@ RUN apt install -y libffi-dev libssl-dev zlib1g-dev libbz2-dev libncurses5-dev \
RUN mkdir -pv /root/.pip \ RUN mkdir -pv /root/.pip \
&& echo "[global]" > /root/.pip/pip.conf \ && echo "[global]" > /root/.pip/pip.conf \
&& echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \ && echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \
&& echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf && echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf \
&& pip install --no-cache-dir wheel
# Install pip package
RUN pip install --no-cache-dir \
numpy \
wheel \
nose \
pytest \
pytest-xdist
# Install cmake (v3.14.1) # Install cmake (v3.14.1)
RUN cd /tmp \ RUN cd /tmp \

64
docker/mindspore-cpu/runtime/Dockerfile Normal file
View File

@ -0,0 +1,64 @@
FROM ubuntu:18.04
MAINTAINER leonwanghui <leon.wanghui@huawei.com>
# Set env
ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5
ENV PATH /usr/local/bin:$PATH
# Install base tools
RUN apt update \
&& DEBIAN_FRONTEND=noninteractive apt install -y \
vim \
wget \
curl \
xz-utils \
net-tools \
openssh-client \
git \
ntpdate \
tzdata \
tcl \
sudo \
bash-completion
# Install compile tools
RUN DEBIAN_FRONTEND=noninteractive apt install -y \
gcc \
g++ \
zlibc \
make \
libgmp-dev \
patch \
autoconf \
libtool \
automake \
flex
# Set bash
RUN echo "dash dash/sh boolean false" | debconf-set-selections
RUN DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
# Install python (v3.7.5)
RUN apt install -y libffi-dev libssl-dev zlib1g-dev libbz2-dev libncurses5-dev \
libgdbm-dev libgdbm-compat-dev liblzma-dev libreadline-dev libsqlite3-dev \
&& cd /tmp \
&& wget https://github.com/python/cpython/archive/v3.7.5.tar.gz \
&& tar -xvf v3.7.5.tar.gz \
&& cd /tmp/cpython-3.7.5 \
&& mkdir -p ${PYTHON_ROOT_PATH} \
&& ./configure --prefix=${PYTHON_ROOT_PATH} \
&& make -j4 \
&& make install -j4 \
&& rm -f /usr/local/bin/python \
&& rm -f /usr/local/bin/pip \
&& ln -s ${PYTHON_ROOT_PATH}/bin/python3.7 /usr/local/bin/python \
&& ln -s ${PYTHON_ROOT_PATH}/bin/pip3.7 /usr/local/bin/pip \
&& rm -rf /tmp/cpython-3.7.5 \
&& rm -f /tmp/v3.7.5.tar.gz
# Set pip source
RUN mkdir -pv /root/.pip \
&& echo "[global]" > /root/.pip/pip.conf \
&& echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \
&& echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf

19
docker/mindspore-gpu/0.1.0-alpha/Dockerfile
View File

@ -4,9 +4,9 @@ MAINTAINER leonwanghui <leon.wanghui@huawei.com>
# Set env # Set env
ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5 ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5
ENV CMAKE_ROOT_PATH /usr/local/cmake-3.14.1
ENV OMPI_ROOT_PATH /usr/local/openmpi-3.1.5 ENV OMPI_ROOT_PATH /usr/local/openmpi-3.1.5
ENV PATH ${PYTHON_ROOT_PATH}/bin:${CMAKE_ROOT_PATH}/bin:${OMPI_ROOT_PATH}/bin:/usr/local/bin:$PATH ENV PATH ${OMPI_ROOT_PATH}/bin:/usr/local/bin:$PATH
ENV LD_LIBRARY_PATH ${OMPI_ROOT_PATH}/lib:$LD_LIBRARY_PATH
# Install base tools # Install base tools
RUN apt update \ RUN apt update \
@ -67,21 +67,6 @@ RUN mkdir -pv /root/.pip \
&& echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \ && echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \
&& echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf && echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf
# Install pip package
RUN pip install --no-cache-dir \
numpy \
wheel \
nose \
pytest \
pytest-xdist
# Install cmake (v3.14.1)
RUN cd /tmp \
&& wget https://github.com/Kitware/CMake/releases/download/v3.14.1/cmake-3.14.1-Linux-x86_64.sh \
&& mkdir -p ${CMAKE_ROOT_PATH} \
&& bash ./cmake-3.14.1-Linux-x86_64.sh --prefix=${CMAKE_ROOT_PATH} --exclude-subdir --skip-license \
&& rm -f /tmp/cmake-3.14.1-Linux-x86_64.sh
# Install openmpi (v3.1.5) # Install openmpi (v3.1.5)
RUN cd /tmp \ RUN cd /tmp \
&& wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.5.tar.gz \ && wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.5.tar.gz \

31
docker/mindspore-gpu/devel/Dockerfile
View File

@ -5,8 +5,8 @@ MAINTAINER leonwanghui <leon.wanghui@huawei.com>
# Set env # Set env
ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5 ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5
ENV CMAKE_ROOT_PATH /usr/local/cmake-3.14.1 ENV CMAKE_ROOT_PATH /usr/local/cmake-3.14.1
ENV OMPI_ROOT_PATH /usr/local/openmpi-3.1.5 ENV PATH ${CMAKE_ROOT_PATH}/bin:/usr/local/bin:$PATH
ENV PATH ${PYTHON_ROOT_PATH}/bin:${CMAKE_ROOT_PATH}/bin:${OMPI_ROOT_PATH}/bin:/usr/local/bin:$PATH ENV LD_LIBRARY_PATH /usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
# Install base tools # Install base tools
RUN apt update \ RUN apt update \
@ -35,9 +35,7 @@ RUN DEBIAN_FRONTEND=noninteractive apt install -y \
autoconf \ autoconf \
libtool \ libtool \
automake \ automake \
flex \ flex
libnccl2=2.4.8-1+cuda10.1 \
libnccl-dev=2.4.8-1+cuda10.1
# Set bash # Set bash
RUN echo "dash dash/sh boolean false" | debconf-set-selections RUN echo "dash dash/sh boolean false" | debconf-set-selections
@ -65,15 +63,8 @@ RUN apt install -y libffi-dev libssl-dev zlib1g-dev libbz2-dev libncurses5-dev \
RUN mkdir -pv /root/.pip \ RUN mkdir -pv /root/.pip \
&& echo "[global]" > /root/.pip/pip.conf \ && echo "[global]" > /root/.pip/pip.conf \
&& echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \ && echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \
&& echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf && echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf \
&& pip install --no-cache-dir wheel
# Install pip package
RUN pip install --no-cache-dir \
numpy \
wheel \
nose \
pytest \
pytest-xdist
# Install cmake (v3.14.1) # Install cmake (v3.14.1)
RUN cd /tmp \ RUN cd /tmp \
@ -81,15 +72,3 @@ RUN cd /tmp \
&& mkdir -p ${CMAKE_ROOT_PATH} \ && mkdir -p ${CMAKE_ROOT_PATH} \
&& bash ./cmake-3.14.1-Linux-x86_64.sh --prefix=${CMAKE_ROOT_PATH} --exclude-subdir --skip-license \ && bash ./cmake-3.14.1-Linux-x86_64.sh --prefix=${CMAKE_ROOT_PATH} --exclude-subdir --skip-license \
&& rm -f /tmp/cmake-3.14.1-Linux-x86_64.sh && rm -f /tmp/cmake-3.14.1-Linux-x86_64.sh
# Install openmpi (v3.1.5)
RUN cd /tmp \
&& wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.5.tar.gz \
&& tar -xvf openmpi-3.1.5.tar.gz \
&& cd /tmp/openmpi-3.1.5 \
&& mkdir -p ${OMPI_ROOT_PATH} \
&& ./configure --prefix=${OMPI_ROOT_PATH} \
&& make -j4 \
&& make install -j4 \
&& rm -rf /tmp/openmpi-3.1.5 \
&& rm -f /tmp/openmpi-3.1.5.tar.gz

80
docker/mindspore-gpu/runtime/Dockerfile Normal file
View File

@ -0,0 +1,80 @@
FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
MAINTAINER leonwanghui <leon.wanghui@huawei.com>
# Set env
ENV PYTHON_ROOT_PATH /usr/local/python-3.7.5
ENV OMPI_ROOT_PATH /usr/local/openmpi-3.1.5
ENV PATH ${OMPI_ROOT_PATH}/bin:/usr/local/bin:$PATH
ENV LD_LIBRARY_PATH ${OMPI_ROOT_PATH}/lib:$LD_LIBRARY_PATH
# Install base tools
RUN apt update \
&& DEBIAN_FRONTEND=noninteractive apt install -y \
vim \
wget \
curl \
xz-utils \
net-tools \
openssh-client \
git \
ntpdate \
tzdata \
tcl \
sudo \
bash-completion
# Install compile tools
RUN DEBIAN_FRONTEND=noninteractive apt install -y \
gcc \
g++ \
zlibc \
make \
libgmp-dev \
patch \
autoconf \
libtool \
automake \
flex \
libnccl2=2.4.8-1+cuda10.1 \
libnccl-dev=2.4.8-1+cuda10.1
# Set bash
RUN echo "dash dash/sh boolean false" | debconf-set-selections
RUN DEBIAN_FRONTEND=noninteractive dpkg-reconfigure dash
# Install python (v3.7.5)
RUN apt install -y libffi-dev libssl-dev zlib1g-dev libbz2-dev libncurses5-dev \
libgdbm-dev libgdbm-compat-dev liblzma-dev libreadline-dev libsqlite3-dev \
&& cd /tmp \
&& wget https://github.com/python/cpython/archive/v3.7.5.tar.gz \
&& tar -xvf v3.7.5.tar.gz \
&& cd /tmp/cpython-3.7.5 \
&& mkdir -p ${PYTHON_ROOT_PATH} \
&& ./configure --prefix=${PYTHON_ROOT_PATH} \
&& make -j4 \
&& make install -j4 \
&& rm -f /usr/local/bin/python \
&& rm -f /usr/local/bin/pip \
&& ln -s ${PYTHON_ROOT_PATH}/bin/python3.7 /usr/local/bin/python \
&& ln -s ${PYTHON_ROOT_PATH}/bin/pip3.7 /usr/local/bin/pip \
&& rm -rf /tmp/cpython-3.7.5 \
&& rm -f /tmp/v3.7.5.tar.gz
# Set pip source
RUN mkdir -pv /root/.pip \
&& echo "[global]" > /root/.pip/pip.conf \
&& echo "trusted-host=mirrors.aliyun.com" >> /root/.pip/pip.conf \
&& echo "index-url=http://mirrors.aliyun.com/pypi/simple/" >> /root/.pip/pip.conf
# Install openmpi (v3.1.5)
RUN cd /tmp \
&& wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.5.tar.gz \
&& tar -xvf openmpi-3.1.5.tar.gz \
&& cd /tmp/openmpi-3.1.5 \
&& mkdir -p ${OMPI_ROOT_PATH} \
&& ./configure --prefix=${OMPI_ROOT_PATH} \
&& make -j4 \
&& make install -j4 \
&& rm -rf /tmp/openmpi-3.1.5 \
&& rm -f /tmp/openmpi-3.1.5.tar.gz