1. GVT-d on Linux*

Intel® GVT-d is an extension of VT-d that provides hardware accelerated I/O virtualization support for Intel® build-in HD graphics - it is one of the GPU virtualization solutions. GVT-d allows Virtual Machine (VM) to call GPU function via I/O directly pass-through. Only one VM can use the GPU at a given time, while other VMs can peform any work that does not require the GPU. If you want require all the VMs to access GPU, you can look into GVT-g and KVMGT.

Following steps will provide the methodology to enable Intel® Media Server Studio R6 (hereafter referred to as MSS R6) on a VT-d environment. This includes the Host and VM installation, as well as a guide to verify successful installation.

2. Preparing

2.0 Hardware requirement

HOST: You will need a CPU that supports Intel® Media Server Studio. You can find the specifications in the Release Notes for Linux* here. In addition, ensure the CPU has support for VT-d, by searching on http://ark.intel.com with your CPU mode as the keyword. In this article, we use the GIGABYTE* GB-BXi7-4770R as the HOST machine.
REMOTE OS: To access the HOST machine remotely, you can install either Microsoft* Windows* or Linux*. In this article, we use PuTTY is used on Windows 8.1 for remote access.

2.1 Setting up BIOS setting on Host

Enable VT-d options in BIOS. Enable legacy boot up options (UEFI boot option maybe cause VM boot failed)

2.2 Setup Host OS

a) Setup CentOS 7.1 OS in "Server with GUI" mode

During installation, please modify the "Software"– "Software Selection" from the default "Base Environment" to "Server with GUI".

b) Setup proxy if your host is under proxy server

Add following line to /etc/yum.conf

proxy=<proxy-server-ip>:<proxy-port>

c) Setup access to internet (or wget)

Add the following lines to ~/.bashrc to get wget access:

export http_proxy=proxy-server-ip:proxy-port
export https_proxy=proxy-server-ip:proxy-port
export ftp_proxy=proxy-server-ip:proxy-port

d) Setup dependence package

# yum groupinstall "Development Tools"
# yum install -y uuid-c++-devel lzo-devel gcc gcc-c++ git patch texinfo python-devel acpica-tools libuuid-devel ncurses-devel glib2 glib2-devel libaio-devel openssl-devel yajl-devel glibc-devel glibc-devel.i686 pixman-devel libpciaccess-devel pciutils-devel pciutils-devel-static pciutils-libs
# yum install -y tigervnc tigervnc-server SDL.i686 bridge-utils
# wget http://mirror.centos.org/centos/6/os/x86_64/Packages/dev86-0.16.17-15.1.el6.x86_64.rpm
# rpm -Uvh dev86-0.16.17-15.1.el6.x86_64.rpm

3. Setup Dom0 and MSS

3.1 Setup MSS R6 generic package

3.1.1 UMD (user mode driver and components) install steps

$ mkdir driver-setup
$ tar xf SDK*/Generic/intel-linux-media-ocl_generic*.tar.gz -C driver-setup
$ cd driver-setup
(as root)
# ./install_media.sh
# reboot

3.1.2 KMD (kernel mode driver) install steps

(as regular user)
$ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.14.5.tar.xz
$ tar -xJf linux-3.14.5.tar.xz
$ cp  /opt/intel/mediasdk/opensource/patches/kmd/3.14.5/intel-kernel-patches.tar.bz2 .
$ tar -xvjf intel-kernel-patches.tar.bz2
$ cd linux-3.14.5
$ for i in ../intel-kernel-patches/*.patch; do patch -p1 < $i; done
$ cp config-3.14.5-VTd.txt .config  ## refer note for config-3.14.5-VTd.txt
$ make -j 8
(as root)
# make modules_install
# make install
# grub2-mkconfig –o /boot/grub2/grub.cfg
# reboot      ## ==> new kernel 3.14.5

Note: Download config-3.14.5-VTd.txt

3.2 Setup Dom0 on Xen

3.2.1 Build Xen from Source

$ git clone git://xenbits.xen.org/xen.git
$ cd xen
$ git checkout -b FOR-4.5.0 RELEASE-4.5.0
$ ./configure
$ make dist
(as root)
# make install

3.2.2 Setup Xen daemon and dynamic libraries

(as root)
# chkconfig --level 35 xencommons on
# chkconfig --level 35 xendomains on

# cd /etc/ld.so.conf.d/
# cat > libxen-x86_64.conf << EOF
/usr/local/lib
EOF
# ldconfig -v
# reboot

3.2.3 Configuring Grub

a) Edit /etc/grub.d/40_custom, and copy new kernel boot entry from /boot/grub2/grub.cfg.

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'CentOS Linux 7 (Core), with Linux 3.14.5-MSSr6+' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.14.5-MSSr6+-advanced-62f16e28-adcb-4a47-925a-f4813854416f' {
        load_video
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos3'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 --hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 --hint='hd0,msdos3'  62f16e28-adcb-4a47-925a-f4813854416f
        else
          search --no-floppy --fs-uuid --set=root 62f16e28-adcb-4a47-925a-f4813854416f
        fi
        linux16 /boot/vmlinuz-3.14.5-MSSr6+ root=UUID=62f16e28-adcb-4a47-925a-f4813854416f ro rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet
        initrd16 /boot/initramfs-3.14.5-MSSr6+.img
}

b) Insert line as below to set up dom 0 memory and virtual CPU core allocation

multiboot /boot/xen.gz dom0_mem=2048M iommu=1 dom0_max_vcpus=2 dom0_vcpus_pin=1

c) Replace "linux16" and "initrd16" with "module", as below

menuentry 'CentOS Linux 7 (Core), with Linux 3.14.5-MSSr6+ Xen Hypervisor' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.14.5-MSSr6+-advanced-62f16e28-adcb-4a47-925a-f4813854416f' {
        load_video
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos3'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos3 --hint-efi=hd0,msdos3 --hint-baremetal=ahci0,msdos3 --hint='hd0,msdos3'  62f16e28-adcb-4a47-925a-f4813854416f
        else
          search --no-floppy --fs-uuid --set=root 62f16e28-adcb-4a47-925a-f4813854416f
        fi

        multiboot /boot/xen.gz dom0_mem=2048M iommu=1 dom0_max_vcpus=8 dom0_vcpus_pin=1
        module /boot/vmlinuz-3.14.5-MSSr6+ root=UUID=62f16e28-adcb-4a47-925a-f4813854416f ro rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet
        module /boot/initramfs-3.14.5-MSSr6+.img
}

d) Update grub menu

(as root)
# grub2-mkconfig -o /boot/grub2/grub.cfg

e) Reboot and select 'CentOS Linux 7 (Core), with Linux 3.14.5-MSSr6+ Xen Hypervisor' in grub menu.

(as root)
# reboot

3.2.3 Check Xen environment

$ lsmod | grep xen
xen_pciback            55503  0
xen_netback            36536  0
xen_blkback            37265  0
xen_gntalloc           13626  0
xen_gntdev             18675  2
xen_evtchn             13033  1
xenfs                  12978  1
xen_privcmd            13243  4 xenfs

(as root)
# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  7852     8     r-----      54.4

# xl info
host                   : centos71.hsw.zhou
release                : 3.14.5-MSSr6+
version                : #1 SMP Tue Sep 1 13:36:53 CST 2015
machine                : x86_64
……

3.3 Check MediaSDK function on Dom0

$ DISPLAY=:0.0 vainfo
libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.35 (libva 1.3.1)
vainfo: Driver version: 16.4.2.1.39163-ubit
vainfo: Supported profile and entrypoints
……

$ ./sample_multi_transcode_drm -hw -i::h264 input.h264 -o::h264 output.h264 -angle 180 -opencl
$ ./sample_encode_drm h264  -hw -i test.yuv -o test.h264 -w 720 -h 480 -la -lad 20

Note: run media functions on dom0 is not recommended. Because all VM will down if dom0 down with some reason.

4. Setup DomU and MSS

4.1 Start Host with "Linux 3.14.5-MSSr6+ Xen Hypervisor" menu entry

4.2 Create and setting virtual xenbr0

a) Find via file name /etc/sysconfig/network-scripts/ifcfg-

b) Enable Xenbr0

(as root)
# br0.sh <NIC> xenbr0        ## br0.sh refer to 7.1

c) Verify xenbr0

$ ifconfig xenbr0
xenbr0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet 10.239.10.16  netmask 255.255.255.0  broadcast 10.239.10.255
        ether fc:aa:14:db:cf:0d  txqueuelen 0  (Ethernet)
        RX packets 136  bytes 14838 (14.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47  bytes 9236 (9.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

4.3 Create virtual machine

a) Create virtual hard disk image

# dd if=/dev/zero of=./image30g.img seek=10000 bs=3M count=1

b) Create VM configure file

An example can be found at /etc/xen/xlexample.hvm

$ cat centos71-no-vtd.hvm
kernel = "hvmloader"
builder = 'hvm'
memory = 3072
vcpus = 6
cpus = "4,5,6,7"
name = "centos-vm1"
vif = [ 'mac=fc:aa:14:db:cf:4e, bridge=xenbr0, model=e1000' ]
disk = [ 'file:/home/lmsdk/setup-vtd/vm/image30g.img,hda,w', 'file:/home/lmsdk/setup-vtd/vm/CentOS-7-x86_64-DVD-1503-01.iso,hdc:cdrom,r' ]
xen_platform_pci=1
device_model_version='qemu-xen'
sdl=0
vnc=1
vnclisten='0.0.0.0'
vncpasswd='intel123'
serial='pty'
tsc_mode="default"
usb=1
keymap='en-us'

c) Start VM OS and setup CentOS 7.1

(as root)
# xl create centos71-no-vtd.hvm
# vncviewer :0   # input password (intel123) setting in VM configure file

While installation, in "Software"– "Software Selection"– "Base Environment", choose "Server with GUI".

d) Setup MSS R6 with Gold configure.

Refer CentOS 7.1 Gold setup in media_server_studio_getting_started_guide.pdf

e) Don’t test Media Functions on VM and close VM

(as root)
# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  2048     8     r-----     977.1
centos-vm1                                   3  3072     6     -b----      39.8

# xl dest 3

5. Set VGA PCI pass-through

5.1 Login host with putty and make sure host has login X window

5.2 Set VGA PCI pass-through on ssh client

(as root)
# xl pci-assignable-add 0000:00:02.0@1c,msitranslate=1
# xl pci-assignable-list
0000:00:02.0

5.3 Modify VM configure with GVT-d

$ cat centos71.hvm
kernel = "hvmloader"
builder = 'hvm'
memory = 2048
vcpus = 2
cpus = ["4", "5"]
name = "centos-vm1"
gfx_passthru=1
pci=['00:02.0@1c,msitranslate=1']
vif = [ 'mac=fc:aa:14:db:cf:4e, bridge=xenbr0, model=e1000' ]
disk = [ 'file:image30g.img,hda,w', 'file:CentOS-7-x86_64-DVD-1503-01.iso,hdc:cdrom,r' ]
xen_platform_pci=1
device_model_version='qemu-xen-traditional'
device_model_override='/usr/local/lib/xen/bin/qemu-dm'
sdl=0
#vnc=1
#vnclisten='0.0.0.0'
#vncpasswd='intel123'
serial='pty'
tsc_mode=0
usb=1
keymap='en-us'

5.4 Boot up VM with GVT-d

(as root)
# xl create centos71.hvm

6. Verify GVT-d on VM

6.1 Verify graphics PCI device

$ lspci –nn | grep VGA
00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:0d22] (rev 08)

6.2 Please look at "Verify correct installation" section in MSS R6 Getting Started Guide to ensure MSS works on VM with root user.

6.3 Verify OpenCL and look-ahead bitrate control work on VM

(as root)
# DISPLAY=:0.0 ./sample_multi_transcode_drm -hw -i::h264 test.h264 -o::h264 output.h264 -angle 180 -opencl
# DISPLAY=:0.0 ./sample_encode_drm h264  -hw -i test.yuv -o test-la-out.h264 -w 720 -h 480 -la -lad 20

7. Script for setup GVT-d

7.1 br0.sh

#!/bin/bash

if [ $# != 2 ] ; then
    echo "Usage: `basename $0` <NIC> <BRIDGE>"
    echo "       NIC: find it via ifconfig"
    echo "       BRIDGE: xenbr0 or br0"
    exit 1;
fi

eth0=$1
br0=$2

ifconfig $eth0 down                         # close eth0 first
brctl addbr $br0                            # add virtual bridge br0
brctl addif $br0 $eth0                      # add interface eth0 on br0
brctl stp $br0 off                          # only on bridge, so close generate tree protocol
brctl setfd $br0 1                          # set br0 delay time of forward
brctl sethello $br0 1                       # set br0 hello time
ifconfig $br0 0.0.0.0 promisc up            # open br0
ifconfig $eth0 0.0.0.0 promisc up           # open eth0
dhclient $br0                               # br0 get IP address from DHCP server
brctl show $br0                             # show virtual bridge list
brctl showstp $br0                          # show interface information on br0