系列文章：

【翻譯】QEMU內(nèi)部機(jī)制：宏觀架構(gòu)和線程模型

【翻譯】QEMU內(nèi)部機(jī)制：vhost的架構(gòu)

【翻譯】QEMU內(nèi)部機(jī)制：頂層概覽(本文)

【翻譯】QEMU內(nèi)部機(jī)制：內(nèi)存

原文地址：http://blog.vmsplice.net/2011/03/qemu-internals-big-picture-overview.html

原文時(shí)間：2011年3月9日

作者介紹：Stefan Hajnoczi來自紅帽公司的虛擬化團(tuán)隊(duì)，負(fù)責(zé)開發(fā)和維護(hù)QEMU項(xiàng)目的block layer, network subsystem和tracing subsystem。

目前工作是multi-core device emulation in QEMU和host/guest file sharing using vsock，過去從事過disk image formats, storage migration和I/O performance optimization

?

QEMU內(nèi)部機(jī)制：頂層概覽

在上一篇【翻譯】QEMU內(nèi)部機(jī)制：宏觀架構(gòu)和線程模型中，我直接介紹了QEMU的線程模型，并沒有提到頂層的架構(gòu)是什么樣的。

本文將對QEMU頂層架構(gòu)進(jìn)行介紹，以幫助理解其線程模型。

vm的運(yùn)行形式

vm是通過qemu程序創(chuàng)建的，一般就是qemu-kvm或kvm程序。在一臺(tái)運(yùn)行有3個(gè)vm的宿主機(jī)上，將看到3個(gè)對應(yīng)的qemu進(jìn)程：

當(dāng)vm關(guān)機(jī)時(shí)，qemu進(jìn)程就會(huì)退出。vm重啟時(shí)，為了方便，qemu進(jìn)程不會(huì)被重啟。不過先關(guān)閉并重新開啟qemu進(jìn)程也是完全可以的。

vm的內(nèi)存

當(dāng)qemu啟動(dòng)時(shí)，vm的內(nèi)存就會(huì)被分配。-mem-path參數(shù)可以支持使用基于文件的內(nèi)存系統(tǒng)，比如說hugetlbfs等。無論如何，內(nèi)存會(huì)映射至qemu進(jìn)程的地址空間，從vm的角度來看，這就是其物理內(nèi)存。

qemu支持大端和小端字節(jié)序架構(gòu)，所以從qemu代碼訪問vm內(nèi)存時(shí)要小心。字節(jié)序的轉(zhuǎn)換不是直接訪問vm的內(nèi)存進(jìn)行的，而是由輔助函數(shù)負(fù)責(zé)。這使得可以從宿主機(jī)上運(yùn)行具有不同字節(jié)序的vm。

kvm虛擬化

kvm是linux內(nèi)核中的虛擬化特性，它使得類似于qemu的程序能夠安全的在宿主cpu上直接執(zhí)行vm的代碼。但是必須得到宿主機(jī)cpu的支持。當(dāng)前，kvm在x86, ARMv8, ppc, s390和MIPS等CPU上都已被支持。

為了使用kvm執(zhí)行vm的代碼，qemu進(jìn)程會(huì)訪問/dev/kvm設(shè)備并發(fā)起KVM_RUN ioctl調(diào)用。kvm內(nèi)核模塊會(huì)借助Intel和AMD提供的硬件虛擬化擴(kuò)展功能執(zhí)行vm的代碼。當(dāng)vm需要訪問硬件設(shè)備寄存器、暫停訪問cpu或執(zhí)行其他特殊操作時(shí)，kvm就會(huì)把控制權(quán)交給qemu。此時(shí)，qemu會(huì)模擬操作的預(yù)期輸出，或者只是在暫停的vm CPU的場景下等待下一次vm的中斷。

vm的cpu執(zhí)行任務(wù)的基本流程如下：
open("/dev/kvm")
ioctl(KVM_CREATE_VM)
ioctl(KVM_CREATE_VCPU)
for (;;) {
ioctl(KVM_RUN)
switch (exit_reason) {
case KVM_EXIT_IO: /* ... */
case KVM_EXIT_HLT: /* ... */
}
}

從宿主機(jī)視角來看vm的執(zhí)行

宿主機(jī)內(nèi)核會(huì)像調(diào)度其他進(jìn)程一樣來調(diào)度qemu進(jìn)程。多個(gè)vm在完全不知道其他vm的存在的情況下同時(shí)運(yùn)行。Firefox或Apache等應(yīng)用程序也會(huì)與qemu競爭宿主機(jī)資源，但可以使用資源控制方法來隔離qemu并使其優(yōu)先級(jí)更高。

qemu在用戶空間進(jìn)程中模擬了一個(gè)完整的虛擬機(jī)，它無從得知vm中運(yùn)行有哪些進(jìn)程。也就是說，qemu為vm提供了RAM、能夠執(zhí)行vm的代碼并且能夠模擬硬件設(shè)備，從而，在vm中可以運(yùn)行任何類型的操作系統(tǒng)。宿主機(jī)無法“窺視”任意vm。

vm會(huì)在每個(gè)虛擬cpu上擁有一個(gè)vcpu線程。一個(gè)獨(dú)立的iothread線程會(huì)通過運(yùn)行select事件循環(huán)來處理類似于網(wǎng)絡(luò)數(shù)據(jù)包或硬盤等IO請求。具體細(xì)節(jié)已經(jīng)在上一篇【翻譯】QEMU內(nèi)部機(jī)制：宏觀架構(gòu)和線程模型中討論過。

下圖是宿主機(jī)視角的qemu進(jìn)程結(jié)構(gòu)：

其他信息

希望本文能夠讓你熟悉qemu和kvm架構(gòu)。
以下是兩則介紹kvm架構(gòu)的ppt，可以了解到更多細(xì)節(jié)：
Jan Kiszka's Linux Kongress 2010： Architecture of the Kernel-based Virtual Machine (KVM)
我自己寫的KVM Architecture Overview

?

?

===精選評(píng)論===
->/dev/kvm設(shè)備是由誰打開的？vcpu線程還是iothread線程？
它是在qemu啟動(dòng)時(shí)由主線程調(diào)用的。請注意，每個(gè)vcpu都有其自己的文件描述符，而/dev/kvm是vm的全局文件描述符，它并非專屬于某個(gè)vcpu。

->vcpu線程如何給出cpu的錯(cuò)覺?它是否需要在上下文切換時(shí)維護(hù)cpu上下文？或者說它是否必須硬件虛擬化技術(shù)的支持？
是的，需要硬件支持。kvm模塊的ioctl接口支持操作vcpu寄存器的狀態(tài)。與物理CPU在機(jī)器開啟時(shí)具有初始寄存器狀態(tài)一樣，QEMU在復(fù)位時(shí)也會(huì)對vcpu進(jìn)行初始化。
KVM需要硬件的支持，intel cpu上稱為VMX，AMD cpu上稱為SVM。二者并不兼容，所以kvm為二者分別提供了代碼上的支持。

->hypervisor和vm如何實(shí)現(xiàn)直接交互？
這要看你是需要什么樣的交互，virtio-serial可以用于任何vm/host的交互。
qemu guest agent程序建立于virtio-serial之上，他支持json rpc api。
它支持宿主機(jī)在vm中調(diào)用一系列命令（比如查詢IP、備份應(yīng)用等）

->能否解釋vm中的一個(gè)應(yīng)用的IO操作流程是怎樣的么？
根據(jù)KVM還是TCG、MMIO還是PIO，ioeventfd是否啟用，會(huì)有多種不同的代碼路徑。
基本流程是vm的內(nèi)存或IO讀寫操作會(huì)“陷入”kvm內(nèi)核模塊，kvm從KVM_RUN的ioctl調(diào)用返回后，將返回值交給QEMU處理。
QEMU找到負(fù)責(zé)該塊內(nèi)存地址的模擬設(shè)備，并調(diào)用其處理函數(shù)。當(dāng)該設(shè)備處理完成之后，QEMU會(huì)使用KVM_RUN的ioctl調(diào)用重入進(jìn)vm。
如果沒有使用Passthrough，那么vm看到的是一個(gè)被模擬出來的網(wǎng)卡設(shè)備，此時(shí)，kvm和qemu不會(huì)直接讀寫物理網(wǎng)卡。他們會(huì)將數(shù)據(jù)包交給linux的網(wǎng)絡(luò)棧(比如tap設(shè)備)進(jìn)行處理。
virtio可以模擬網(wǎng)絡(luò)、存儲(chǔ)等虛擬設(shè)備，它會(huì)用一些優(yōu)化方式，但是基本原理還是模擬一個(gè)“真實(shí)”設(shè)備。

->kvm內(nèi)核模塊是如何捕捉到virtqueue的kick動(dòng)作的？
ioeventfd被注冊為KVM的IO總線設(shè)備，kvm使用ioeventfd_write()通知ioeventfd。
trace流程為：
vmx_handle_exit with EXIT_REASON_IO_INSTRUCTION
--> handle_io
--> emulate_instruction
--> x86_emulate_instruction
--> x86_emulate_insn
--> writeback
--> segmented_write
--> emulator_write_emulated
--> emulator_read_write_onepage
--> vcpu_mmio_write
--> ioeventfd_write
該流程顯示了當(dāng)vm kicks宿主機(jī)時(shí)，信號(hào)是如何通知ioeventfd。

?

?

原文如下：

QEMU Internals: Big picture overview

Last week I started the?QEMU Internals?series to share knowledge of how QEMU works. I dove straight in to the?threading model?without a high-level overview. I want to go back and provide the big picture so that the details of the threading model can be understood more easily.

The story of a guest

A guest is created by running the?qemu?program, also known as?qemu-kvm?or just?kvm. On a host that is running 3 virtual machines there are 3?qemu?processes:


When a guest shuts down the?qemu?process exits. Reboot can be performed without restarting the?qemu?process for convenience although it would be fine to shut down and then start?qemu?again.

Guest RAM

Guest RAM is simply allocated when?qemu?starts up. It is possible to pass in file-backed memory with?-mem-pathsuch that?hugetlbfs?can be used. Either way, the RAM is mapped in to the?qemu?process' address space and acts as the "physical" memory as seen by the guest:


QEMU supports both big-endian and little-endian target architectures so guest memory needs to be accessed with care from QEMU code. Endian conversion is performed by helper functions instead of accessing guest RAM directly. This makes it possible to run a target with a different endianness from the host.

KVM virtualization

KVM is a virtualization feature in the Linux kernel that lets a program like?qemu?safely execute guest code directly on the host CPU. This is only possible when the target architecture is supported by the host CPU. Today KVM is available on x86, ARMv8, ppc, s390, and MIPS CPUs.
In order to execute guest code using KVM, the?qemu?process opens?/dev/kvm?and issues the?KVM_RUN?ioctl. The KVM kernel module uses hardware virtualization extensions found on modern Intel and AMD CPUs to directly execute guest code. When the guest accesses a hardware device register, halts the guest CPU, or performs other special operations, KVM exits back to?qemu. At that point?qemu?can emulate the desired outcome of the operation or simply wait for the next guest interrupt in the case of a halted guest CPU.
The basic flow of a guest CPU is as follows:
open("/dev/kvm") ioctl(KVM_CREATE_VM) ioctl(KVM_CREATE_VCPU) for (;;) {ioctl(KVM_RUN)switch (exit_reason) {case KVM_EXIT_IO: /* ... */case KVM_EXIT_HLT: /* ... */} }

The host's view of a running guest

The host kernel schedules?qemu?like a regular process. Multiple guests run alongside without knowledge of each other. Applications like Firefox or Apache also compete for the same host resources as?qemu?although resource controls can be used to isolate and prioritize?qemu.

Since?qemu?system emulation provides a full virtual machine inside the?qemu?userspace process, the details of what processes are running inside the guest are not directly visible from the host. One way of understanding this is that?qemu?provides a slab of guest RAM, the ability to execute guest code, and emulated hardware devices; therefore any operating system (or no operating system at all) can run inside the guest. There is no ability for the host to peek inside an arbitrary guest.

Guests have a so-called?vcpu?thread per virtual CPU. A dedicated?iothread?runs a?select(2)?event loop to process I/O such as network packets and disk I/O completion. For more details and possible alternate configuration, see the?threading model?post.

The following diagram illustrates the?qemu?process as seen from the host:

Further information

Hopefully this gives you an overview of QEMU and KVM architecture. Feel free to leave questions in the comments and check out other?QEMU Internals?posts for details on these aspects of QEMU.

Here are two presentations on KVM architecture that cover similar areas if you are interested in reading more:

Jan Kiszka's Linux Kongress 2010 presentation on the?Architecture of the Kernel-based Virtual Machine (KVM). Very good material.

My own attempt at presenting a?KVM Architecture Overview?from 2010.

轉(zhuǎn)載于:https://www.cnblogs.com/qxxnxxFight/p/11058132.html

總結(jié)

以上是生活随笔為你收集整理的【翻译】QEMU内部机制：顶层概览的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。