深度学习研究笔记-caffe安装

2016年04月05日

说到深度学习,不得不说大名鼎鼎的开源项目caffe,今天晚上就折腾,在iMac上安装caffe,下面我会对所有步骤进行记录,以便后期整理。
开始安装了,先看官网学习一下http://caffe.berkeleyvision.org/ Caffe是Yangqing Jia在伯克利读博士时创立的,好牛啊!然后有数以千计的开发者为该项目做贡献,我也想成为之一! Caffe从架构,代码,速度,社区等几个方面都有优势,所以深度学习首选Caffe

接下来我们安装试试,先说下,我要在iMac上安装,iMac上是AMD的显卡,GPU资源看来是不能用了,看看以后能不能找老婆申请买个家用服务器,现在先不用GPU安装。安装文档在这里http://caffe.berkeleyvision.org/installation.html 需要安装一些前置的依赖,首先是CUDA,这个真是臣妾做不到啊,AMD显卡伤不起只能略过。 下面安装BLAS,BLAS是什么呢?BLAS,即基础线性代数子程序库,里面拥有大量已经编写好的关于线性代数运算的程序。这么好的轮子得赶快安装,但是接下来非常纠结,文档中介绍BLAS可以通过三个实现安装分别是 ATLAS,MKL和OpenBLAS,无论选哪个都要知道区别啊,看来需要先搜索一番,看了知乎的一篇文章https://www.zhihu.com/question/27872849,大概又研究了七八分钟,有了初步的印像,ATLAS名字很霸气,MKL对英特尔CPU做了优化,OpenBLAS是中国人开发的,这时爱国情怀涌了上来,首先给中国人开发的OpenBLAS机会,用这个。来到项目地址https://github.com/xianyi/OpenBLAS,按惯例先看readme,看起来安装很方便啊,先下代码 git clone git://github.com/xianyi/OpenBLAS.git 这个时候感觉用sublime写markdown不爽啊,就比如上面的命令行不知道啥效果,赶紧切换到macdown去写,切过去发现macdown需要升级,完美主义心又出来了,得升级一下,趁着升级赶紧过来记录一下这时细腻的心情。
接下来直接make,顶配iMac竟然也需要make好几分钟,然后悲剧的是error了

ld: ldlibrary ldnot:  found:library   forlibrary not - notlcrt1.10.6.ofound 
 foundfor  for- -lcrt1.10.6.o
lcrt1.10.6.o
collect2: error: ld returned 1 exit status
collect2: error: ld returned 1 exit status
collect2: error: ld returned 1 exit status
ld: library not found for -lcrt1.10.6.o
collect2: error: ld returned 1 exit status
make[2]: *** [testslamch] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [testdlamch] Error 1
make[2]: *** [testlsame] Error 1
make[2]: *** [testsecond] Error 1
make[1]: *** [lapack_install] Error 2
make: *** [netlib] Error 2

貌似是链接错误,但是我首先是花了一点儿时间整上面的代码高亮,接下来不能看源码查找错误,这样速度太慢,今天睡不了觉了,先用openblas+mac的关键字google一下,看到了这片文章http://stackoverflow.com/questions/21067723/how-to-build-openblas-on-osx-mavericks,原来用brew也可以装,试试看。brew install homebrew/science/openblas,安装成功了,这时我想起了《三体》里面一句话,前进,前进,不择手段的前进,我的目标是学习Deep learning,安装完OpenBLAS就得了,得把安装过程及结果信息记录下来,感觉后面有几率用到

lizhedeiMac:OpenBLAS lizhe$ brew install homebrew/science/openblas
Warning: You are using OS X 10.11.
We do not provide support for this pre-release version.
You may encounter build failures or other breakage.
==> Installing openblas from homebrew/homebrew-science
==> Using Homebrew-provided fortran compiler.
This may be changed by setting the FC environment variable.
==> Downloading https://github.com/xianyi/OpenBLAS/archive/v0.2.14.tar.gz
==> Downloading from https://codeload.github.com/xianyi/OpenBLAS/tar.gz/v0.2.14
######################################################################## 100.0%
==> make FC=/usr/local/bin/gfortran libs netlib shared
==> make FC=/usr/local/bin/gfortran tests
==> make PREFIX=/usr/local/Cellar/openblas/0.2.14_1 install
==> Caveats
This formula is keg-only, which means it was not symlinked into /usr/local.

OS X already provides this software and installing another version in
parallel can cause all kinds of trouble.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/openblas/lib
    CPPFLAGS: -I/usr/local/opt/openblas/include

==> Summary
🍺  /usr/local/Cellar/openblas/0.2.14_1: 14 files, 42M, built in 8.4 minutes

接下来,我们安装Boost,我记得Boost早就安装完了确认一下brew install boost,果然安装完了,版本也符合要求。

lizhedeiMac:OpenBLAS lizhe$ brew install boost
Warning: boost-1.58.0 already installed
Warning: You are using OS X 10.11.
We do not provide support for this pre-release version.
You may encounter build failures or other breakage.

下面是安装一些可选的依赖,OpenCV安装完了,其他的不装了。 对于Python Caffe,需要安装一些程序,For Python Caffe: Python 2.7 or Python 3.3+, numpy (>= 1.7), boost-provided boost.python,其他都装完了,除了这个brew install boost-python

下一步需要试着安装caffe了,先下载release代码,地址在这https://github.com/BVLC/caffe/tags, 修改makefile文件,打开只使用CPU模式,变更BLAS的实现BLAS := OpenBlas,大致看了下差不多了,先试编一下,不行再修改,直接make all,然后果不其然就error了,

lizhedeiMac:caffe-rc3 lizhe$ make all
No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.
/bin/sh: line 0: [: -gt: unary operator expected
PROTOC src/caffe/proto/caffe.proto
CXX .build_release/src/caffe/proto/caffe.pb.cc
CXX src/caffe/blob.cpp
In file included from src/caffe/blob.cpp:7:
In file included from ./include/caffe/util/math_functions.hpp:11:
./include/caffe/util/mkl_alternate.hpp:11:10: fatal error: 'cblas.h' file not found
#include <cblas.h>
         ^
1 error generated.
make: *** [.build_release/src/caffe/blob.o] Error 1

时间有点晚了,需要睡觉,但是我要把这个error改了,应该修改makefile文件就行,之前安装openblas保留下来的信息立功了,修改openblas文件路径就可以了

BLAS_INCLUDE := /usr/local/opt/openblas/include
BLAS_LIB := /usr/local/opt/openblas/lib

接下来再次运行make all,这个error过了,但是有遇到了另一个error

。。。
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_tanh_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn.o) has no symbols
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
clang: warning: argument unused during compilation: '-pthread'
ld: framework not found vecLib
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1

预料中的得荡起伏啊,google了一番,找到思路了,需要修改FindvecLib.cmake文件,只能明天再改,先睡觉了。

又回来了,先从vecLib这个framework入手,看文档发现这个库已经失效了https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/OSXTechnologyOverview/SystemFrameworks/SystemFrameworks.html vecLib.framework 10.0 N/A Deprecated. Use Accelerate.framework instead. See Accelerate Framework. 现在需要用Accelerate Framework了,再仔细看了下FindvecLib.cmake文件,发现已经处理这种case了,看来还是要想办法修改FindvecLib.cmake,里面的xcode路径明显错了,目前系统版本是10.11,上面写的还是10.9,改成这样

find_path(vecLib_INCLUDE_DIR vecLib.h
          DOC "vecLib include directory"
          PATHS /System/Library/${__veclib_include_suffix}
                /System/Library/Frameworks/Accelerate.framework/Versions/Current/${__veclib_include_suffix}
                /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers)

再执行make all情况如下

lizhedeiMac:caffe-rc3 lizhe$ make all
No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.
/bin/sh: line 0: [: -gt: unary operator expected
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
clang: warning: argument unused during compilation: '-pthread'
ld: framework not found vecLib
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1

又出了一个新错误,没有安装xcode命令行工具,赶紧装一个xcode-select --install,趁着安装的功夫说一下,我和老婆打算去包一块农场种菜,当时是别人种,我们每周去摘菜,周末去看看。 xcode命令行工具装完了,再次make all,竟然成功了,然后运行下make test也成功了,紧接着安装py-caffe,make pycaffe 出现错误

lizhedeiMac:caffe-rc3 lizhe$ make pycaffe
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
python/caffe/_caffe.cpp:10:10: fatal error: 'numpy/arrayobject.h' file not found
#include <numpy/arrayobject.h>
         ^
1 error generated.
make: *** [python/caffe/_caffe.so] Error 1

尝试重新安装numpy,brew uninstall numpy然后brew install numpy,输出结果如下

Python modules have been installed and Homebrew's site-packages is not
in your Python sys.path, so you will not be able to import the modules
this formula installed. If you plan to develop with these modules,
please run:
  mkdir -p /Users/lizhe/Library/Python/2.7/lib/python/site-packages
  echo 'import site; site.addsitedir("/usr/local/lib/python2.7/site-packages")' >> /Users/lizhe/Library/Python/2.7/lib/python/site-packages/homebrew.pth
==> Summary
🍺  /usr/local/Cellar/numpy/1.9.2_1: 692 files, 14M, built in 64 seconds

乖乖照着上面的提示配置一下,再试试make pycaffe,问题依旧。尝试了各种姿势安装numpy,依旧不行,然后找到了这篇文章https://github.com/rainyear/lolita/issues/10 如果是通过Homebrew安装的numpy,需要添加numpy路径,否则会报错:

python/caffe/_caffe.cpp:10:10: fatal error: 'numpy/arrayobject.h' file not found

# Add this to PYTHON_INCLUDE in Makefile.config
PYTHON_INCLUDE := /usr/include/python2.7 \
    /usr/local/Cellar/numpy/1.9.2_1/lib/python2.7/site-packages/numpy/core/include

遗漏了这个细节,赶快去改caffe配置文件,再次make pycaffe 顺利完成。 下一步的目标是在python中可以调用caffe,但是在安装python包时遇到了各种问题,因为Sorry, Unix fans: OS X El Capitan kills root。。。。http://www.infoworld.com/article/2988096/mac-os-x/sorry-unix-fans-os-x-el-capitan-kills-root.html 得想想办法,对于个人用户来说,可以这样安装python包 for req in $(cat requirements.txt); do pip install --user $req; done 参考的这个文章:http://apple.stackexchange.com/questions/209572/how-to-use-pip-after-the-el-capitan-max-os-x-upgrade 最后测试一下

python
import caffe

终于安装成功了,用了两个晚上,各种惊心动魄,各种绝望,虽然安装成功了,但是caffe能做什么呢?我也不知道,下面得运行几个demo试试了,明天再说吧。