Creating virtual machiness on Linux --- the hard way

9月 17 2019 14 分钟读完 (约 2128 字)

Two ways of using KVM-based vms on Linux are introduced. One is using virt-manager GUI, another is using the raw qemu-system.

I. Prerequisites

1 2	pacman -S qemu bridge-utils iptables dnsmasq ebtables pacman -S libvirt

check for module loading:

1 2	lsmod \| grep kvm lsmod \| grep virtio

To auto load a module at boot, create a file like ‘/etc/modules-load.d/virtio.conf’

1 2	# Load virtio at boot virtio

II. Virt Manager

1	pacman -S virt-manager

1 2	sudo usermod --append --groups libvirt $(whoami) sudo systemctl start libvirtd

check for nat:

1 2	sudo virsh net-list --all # sudo virsh net-start default

Then install using GUI.
if you want a raw image, create the img file by yourself

1	qemu-img create -f raw imgfile 20G

II.A Win7 VM installation and usage

aur install virtio-win
pacman install dmidecode

Under hard disk OS-images directory, find the file windows7_Ultimate_x64.iso and copy it into /var/lib/libvirt/images.
This is a ghost OS image with some preinstall software. I use this because the activation tool is inside.

Just put the script and Windows Loader.ext in the same directory and then run the script in Administrator.

Real time clock

In virt manager, enable XML editing.
Then replace <clock offset='utc'> with <clock offset='localtime'>
Administrator open cmd:

1	bcdedit /set {default} USEPLATFORMCLOCK on

Office 2016 pack

mount the iso file: Office_2016_x64_ZH_16.0.7466.2038.iso
off-line installer:

start one of the Office app
convert and activate

QXL Acceleration

1 2	lsmod\|grep qxl # check sudo modprobe qxl bochs_drm # if not mod

To auto load a module at boot, create a file like ‘/etc/modules-load.d/qxl.conf’

1
2
3

# Load qxl at boot
qxl
bochs_drm

This is not enough. Because we just build a base on the server, but do nothing on the Windows guest. Look at this:

Next install spice-guest-tools-latest.exe downloaded from https://www.spice-space.org/download.html.

done installed qxl driver

比较不好的一个bug是虚拟机内部的复制粘贴不好用了，所以对于跨机的复制粘贴，还是采用原始的文件方式来实现。

make sure sshd on host is on
open one term on host and type tty to get device name，say /dev/pts/0

ssh to host using PuTTY from win7 client
echo "copied text" >> /dev/pts/0

You will see “copied text” on the host terminal.

II.B Win10 VM Installation

Because of two bugs of Win7 VM: the first is that copy-and-pasteing for office object always throws an error in VM; the second is that Office2016 on win7 cannot open remote-location file. I change to win10 vm, and I found that’s cool.

II.B.a Launch VM as a desktop program

Go to Virt-manager > Connection Details to find Libvirt URI, here mine is qemu:///system

libvirt-uri

Then open virtual machine, go to View > Details to find the name of the virtual machine, here mine is win10

vm-name

Now we can test the direct-open command:

1	virt-manager --connect qemu:///system --show-domain-console win10

Then create your desktop file under ~/.local/share/applications:

[Desktop Entry]
Version=1.0
Type=Application
Name=win10
Comment=Start Win10 VM
Exec=virt-manager --connect qemu:///system --show-domain-console win10
Icon=your/logo/position/windows.svg
Terminal=false
StartupNotify=false

II.B.b Expand block image size

To add 5G to my win10 raw format volume, first check the state:

# virsh list --all
 Id   Name    State
------------------------
 -    win10   shut off

Then go to increase

1
2
3

# cd /var/lib/libvirt/images/
# ls -lh win10vol.img
# qemu-img resize win10vol.img +5G

Then start win10 VM and right click ‘Start Menu’ -> Disk management -> Extend Volumn C:\

ok.

II.B.c PCI硬件设备直通

Prerequisite: Enable IOMMU(Input/Output Memory Management Unit)
Intel CPU上开启IOMMU需要VT-d的支持。VT-d(Intel Virtualization Technology for Direct I/O)是一个位于CPU、内存和I/O设备之间的硬件设备，它可以拦截I/O设备产生的中断请求和通过DMA方式对虚拟机内存的访问，然后重定向到通过虚拟机监视器（VMM）设定的虚拟机中。VT-d技术支持了虚拟机内部驱动程序直接与硬件设备通信，替代了传统的软件模拟方式，大大提升了虚拟化的I/O性能。

开启方式可分为临时开启和写配置文件开启两种方式：
方式一（临时开启方法）：在计算机启动的grub页面按e进入kernal模式，在启动vmlinuz的参数后面添加 intel_iommu=on 和 iommu=pt

这种开启方法在重启之后配置又会变成原来的样子。

方式二（写配置文件开启方法）：

1 2	su vim /etc/default/grub

在 GRUB_CMDLINE_LINUX_DEFAULT 参数中添加 intel_iommu=on iommu=pt

GRUB_CMDLINE_LINUX_DEFAULT=”loglevel=3 intel_iommu=on iommu=pt”

1	grub-mkconfig -o /boot/grub/grub.cfg

1. 配置无线网卡直通
默认的虚拟机使用NAT模式上网，对一些其他上网模式也是支持的。目前我对于桥接模式的配置没有成功，所以这里记录只成功了的网卡直通模式。
在该模式下，虚拟机直接使用无线网卡联网，宿主机可以用有线连接，当时配置该模式只是为了暂时同步一下微信聊天记录（因为必须在同一wifi下）

成功开机后root使用下面命令检查无线网卡的pci地址：

shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

我的是

IOMMU Group 15:
3a:00.0 Network controller [0280]: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter [168c:003e] (rev 32)

再检查一下地址是否正确和是否支持iommu

1 2	lspci -n -s 3a:00.0 dmesg \| grep -i iommu \| grep 3a:00

显示是支持的：

[ 0.452590] pci 0000:3a:00.0: Adding to iommu group 15

将无线网卡与宿主机解绑：

1	echo "0000:3a:00.0" > /sys/bus/pci/devices/0000\:3a\:00.0/driver/unbind

然后，打开虚拟机软件，添加pci device，与虚拟机绑定:

参考：https://www.cnblogs.com/xia-dong/p/11542771.html

2. 配置USB直通
一般情况下，QEMU的USB软件重定向方式可以满足U盘等设备的使用需求，但是在一些需要特定驱动的方式下（比如无线投影、USB转VGA），需要开启USB直通功能。

[root@arch wsd]# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0c45:670c Microdia Integrated Webcam HD
Bus 001 Device 002: ID 0cf3:e300 Qualcomm Atheros Communications QCA61x4 Bluetooth 4.0
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

通过lsusb命令可以看到，USB接口和蓝牙、摄像头都挂在USB Bus 001下。

检查USB设置是否支持RESET功能：

[root@arch wsd]# for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done
IOMMU group 7
[RESET] 00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #5 [8086:9d14] (rev f1)
IOMMU group 5
        00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-LP CSME HECI #1 [8086:9d3a] (rev 21)
IOMMU group 13
[RESET] 3c:00.0 Non-Volatile memory controller [0108]: SK hynix PC401 NVMe Solid State Drive 256GB [1c5c:1527]
IOMMU group 3
        00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller [8086:9d2f] (rev 21)
        00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Thermal subsystem [8086:9d31] (rev 21)
IOMMU group 11
[RESET] 3a:00.0 Network controller [0280]: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter [168c:003e] (rev 32)
IOMMU group 1
[RESET] 00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics 620 [8086:5917] (rev 07)
IOMMU group 8
[RESET] 00:1c.5 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 [8086:9d15] (rev f1)
IOMMU group 6
[RESET] 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #1 [8086:9d10] (rev f1)
IOMMU group 4
        00:15.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 [8086:9d60] (rev 21)
        00:15.1 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #1 [8086:9d61] (rev 21)
IOMMU group 12
[RESET] 3b:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader [10ec:525a] (rev 01)
IOMMU group 2
        00:04.0 Signal processing controller [1180]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem [8086:1903] (rev 08)
IOMMU group 10
        00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point LPC Controller/eSPI Controller [8086:9d4e] (rev 21)
        00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
        00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-LP HD Audio [8086:9d71] (rev 21)
        00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
IOMMU group 0
        00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5914] (rev 08)
IOMMU group 9
[RESET] 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #9 [8086:9d18] (rev f1)

每行前面带有RESET字样就是可以支持的，可以看到”00:14.0 USB controller”是不支持RESET的。

II.B.d CPU绑定

通过 virsh vcpuinfo win10 可以查看当前使用的CPU情况

wsd@xps13 ~ % sudo virsh vcpuinfo win10                                                                                                                                                                         [1]
VCPU:           0
CPU:            5
State:          running
CPU time:       45.5s
CPU Affinity:   yyyyyyyy

VCPU:           1
CPU:            1
State:          running
CPU time:       42.7s
CPU Affinity:   yyyyyyyy

通过virsh edit win10，在<vcpu placement='static'>2</vcpu>下一行添加：

<cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='7'/>
</cputune>

将虚拟机两个vCPU分别绑定在宿主机6和7号CPU上
再次检查CPU使用情况：

[root@xps13 wsd]# virsh vcpuinfo win10
VCPU:           0
CPU:            6
State:          running
CPU time:       165.3s
CPU Affinity:   ------y-

VCPU:           1
CPU:            7
State:          running
CPU time:       105.4s
CPU Affinity:   -------y

II.B.e 配置双屏

安装virt-viewer

1	sudo pacman -Syu virt-viewer

在Win10虚机配置页新增一个QXL Video，然后启动虚机。
通过remote-viewer登录：

1	remote-viewer spice://localhost:5900

（可能还需要在Windows系统中Ctrl+P设置拓展屏）

II.B.f 单独设置VRAM

默认的QXL内存分配是65535 KB，即64MB，对PhotoShop这样的需要512MB VRAM的应用来说明显不足。
因此使用 virsh edit win10 手动编辑下ram和vram的值：

<video>
    <model type='qxl' ram='524288' vram='524288' vgamem='16384' heads='1' primary='yes'/>
    <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>

设置后直接无法启动了，好奇怪。。

References

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

#Linux #VM #Win10 #qemu #kvm