libvirt + windows 11 + hyper-v 삽질기

일단, 그냥 windows 11 동작은 쉽지만, windows 11 에서 Hyper-v (또는 WSL2) 을 활성화 시키면 무한 재부팅 현상이 발생한다.

libvirt 에서 Windows 11 + Hyper-V 을 동작시키기 위한 구성에 대한 삽질기이다.

Host 정보

• CPU : i9-9900 (Coffee Lake)

CPU: host-passthrough

https://ubuntuforums.org/showthread.php?t=2502573&p=14213272#post14213272

아래의 구성으로 부팅은 되지만...

  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <runtime state="on"/>
      <synic state="on"/>
      <stimer state="on">
        <direct state="on"/>
      </stimer>
      <reset state="on"/>
      <ipi state="on"/>
      <evmcs state="on"/>
    </hyperv>
    <smm state="on"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <feature policy="require" name="vmx"/>
    <feature policy="disable" name="hypervisor"/>
    <feature policy="disable" name="waitpkg"/>
  </cpu>

Hypervisor launch failed; The hypervisor was unable to initialize successfully (phase 0x7), and was not started. This initialization failure may be the result of a platform configuration or firmware issue.

이런 오류가 발생하면서 Hyper-V 는 동작하지 않는다...

CPU: Skylake-Client-noTSX-IBRS 일 때

일단 기본 구성은 아래와 같이 시작

  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv mode="passthrough">
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <smm state="on"/>
    <ioapic driver="qemu"/>
  </features>
  <cpu mode="custom" match="exact" check="partial">
    <model fallback="allow">Skylake-Client-noTSX-IBRS</model>
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <feature policy="require" name="vmx"/>
    <feature policy="disable" name="hypervisor"/>
  </cpu>

IOMMU 사용 불가

    <iommu model="intel">
      <driver intremap="on"/>
    </iommu>

이 cpu 구성에서는 iommu 가 있으면 동작이 불가하다.

성능 문제

• 동작하지만 눈에 띄는 성능 저하가 있다.
• 로그인 창에서 멈추기도 함..

vmx-xsaves 추가

https://bugzilla.redhat.com/show_bug.cgi?id=1974560 이슈를 찾았다.

    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='vmx-xsaves'/>

참고로 이 구성은 아래와 동일하다.

 <cpu mode='custom' match='exact' check='full'>
    <model fallback='allow'>Skylake-Client</model>
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <feature policy="require" name="vmx"/>
    <feature policy="disable" name="hypervisor"/>
    <feature policy='disable' name='hle'/> <!-- noTSX -->
    <feature policy='disable' name='rtm'/> <!-- noTSX -->
    <!-- vmx-xsaves 은 Skylake-Client 에 있음 -->
    <feature policy="disable" name="mpx"/>
  </cpu>

• 하지만 로그인 창에서 무언가 crash 가 발생해 즉시 재부팅한다.
• 다시 부팅하니 동작은 한다. Hyper-V 와 VBS 도 동작한다.
• 하지만 느려서 못쓴다..

virsh capabilities 이용

virsh capabilities 을 이용해 출력 된 cpu 설정은 다음과 같다

    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Client-noTSX-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='248'/>
      <signature family='6' model='158' stepping='12'/>
      <counter name='tsc' frequency='3599999000' scaling='no'/>
      <topology sockets='1' dies='1' cores='8' threads='2'/>
      <maxphysaddr mode='emulate' bits='39'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='osxsave'/>
      <feature name='tsc_adjust'/>
      <feature name='sgx'/>
      <feature name='clflushopt'/>
      <feature name='intel-pt'/>
      <feature name='sgxlc'/>
      <feature name='md-clear'/>
      <feature name='stibp'/>
      <feature name='flush-l1d'/>
      <feature name='arch-capabilities'/>
      <feature name='ssbd'/>
      <feature name='xsaves'/>
      <feature name='sgx1'/>
      <feature name='sgx-edeccssa'/>
      <feature name='sgx-debug'/>
      <feature name='sgx-mode64'/>
      <feature name='sgx-provisionkey'/>
      <feature name='sgx-tokenkey'/>
      <feature name='sgx-aex-notify'/>
      <feature name='pdpe1gb'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='skip-l1dfl-vmentry'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>

여기서 microcode, signature, counter 은 삭제하고 vmx, hypervisor 을 추가했다.

최종

  <cpu mode="custom" match="exact" check="partial">
    <model fallback="allow">Skylake-Client-noTSX-IBRS</model>
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <maxphysaddr mode='emulate' bits='39'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='osxsave'/>
      <feature name='tsc_adjust'/>
      <feature name='sgx'/>
      <feature name='clflushopt'/>
      <feature name='intel-pt'/>
      <feature name='sgxlc'/>
      <feature name='md-clear'/>
      <feature name='stibp'/>
      <feature name='flush-l1d'/>
      <feature name='arch-capabilities'/>
      <feature name='ssbd'/>
      <feature name='xsaves'/>
      <feature name='sgx1'/>
      <feature name='sgx-edeccssa'/>
      <feature name='sgx-debug'/>
      <feature name='sgx-mode64'/>
      <feature name='sgx-provisionkey'/>
      <feature name='sgx-tokenkey'/>
      <feature name='sgx-aex-notify'/>
      <feature name='pdpe1gb'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='skip-l1dfl-vmentry'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
      <!-- added -->
      <feature policy="require" name="aes"/>
      <feature policy="disable" name="hypervisor"/>
  </cpu>

• 빨라졌지만, VBS, Hyper-V 는 실행 불가하다. passthrough 랑 다를게 없다.

osxsave (AVX), invtsc 추가

virsh capabilities 을 통해서 기본적으로 가능한 features 와 maxphysaddr 을 찾은 다음에 아래와 같이 수정했다.

  <cpu mode="custom" match="exact" check="full">
    <model fallback="allow">Skylake-Client</model>
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <maxphysaddr mode="emulate" bits="39"/>
    <feature policy="require" name="vmx"/>
    <feature policy="disable" name="hypervisor"/>
    <feature policy="disable" name="hle"/> <!-- noTSX -->
    <feature policy="disable" name="rtm"/> <!-- noTSX -->
    <feature policy="disable" name="mpx"/> <!-- 지원안함 -->
    <feature policy="require" name="osxsave"/>
    <feature policy="require" name="invtsc"/>
  </cpu>

• invtsc 으로 약간 빨라졌다! (하지만 여전히 큰 성능저하는 있다)
• avx 사용 시 osxsave 도 필요한 듯 하다. 이 때문인지 crash 도 나지 않았다.
• 드디어 VBS, Hyper-V 실행 가능!

l3 cache 추가

  <cpu mode="custom" match="exact" check="full">
    <model fallback="allow">Skylake-Client</model>
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <cache level="3" mode="emulate"/>
    <feature policy="require" name="vmx"/>
    <feature policy="disable" name="hypervisor"/>
    <feature policy="disable" name="hle"/> <!-- noTSX -->
    <feature policy="disable" name="rtm"/> <!-- noTSX -->
    <feature policy="disable" name="mpx"/> <!-- 지원안함 -->
    <feature policy="require" name="osxsave"/>
    <feature policy="require" name="invtsc"/>
  </cpu>

-- l3 cache 추가로 조금 더 개선이 되었다.

결론

답이없다.. 성능 저하 없이 쓰는건 불가하다..
다만 아래 설정 + VBS, HECI 비활성화 하니 일반적인 사용은 가능해졌고, Nested VM (WSL, Hyper-V) 안의 OS 느린 건 어쩔 수 없다.

(VBS 을 켰을 때 느린 이유는 VBS 자체가 지금 보고 있는 OS을 가상으로 돌리기 때문이다)

 

  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv mode="passthrough">
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <smm state="on"/>
    <ioapic driver="qemu"/>
  </features>
  <cpu mode="custom" match="exact" check="full">
    <model fallback="allow">Skylake-Client-noTSX-IBRS</model>
    <topology sockets="1" dies="1" cores="8" threads="1"/>
    <cache level="3" mode="emulate"/>
    <maxphysaddr mode="emulate" bits="39"/>
    <feature policy="require" name="vmx"/>
    <feature policy="require" name="osxsave"/>
    <feature policy="require" name="pdpe1gb"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="disable" name="hypervisor"/>
    <feature policy="disable" name="mpx"/>
  </cpu>