Tweet
I've made alot of progress on getting a Windows VM to have control over a 290x using Xen, but it's not yet working properly. I have a ton of details I can share on what I've done, but for now I have some specific questions for those who have gone through these steps successfuly.
My 290x doesn't seem to support FLR (function level reset). I'm not sure if this is a requirement of all GPUs that will be passed to a VM or just Nvidia cards, but regardless I found this out by running sudo lspci -vv looking for my 290x and then finding the section starting with 'DevCap' (FLReset- is present). For what it's worth, all my devices that report on this functionality are listed as no support (FLReset-). First I need to ensure this is critical to successful PCI passthrouth. If so, then this makes me think I need to either 1) enable something in the BIOS or 2) change a kernel or module parameter.
Here is the output of of xl dmesg:
Here is my /etc/xen/vwin.hvm (vm config):
When I successfully start the VM, here is the output:
The above at first made me think there is an issue, but I not sure that it is at this point. When Windows boots I see the 290x, and when I run the AMD test to see what hardware I have it finds it as being a 290x. I can install the drivers for the card and no errors are reported. I reboot and look at Device Manager and I see the following message associated with the problem 290x device:
Here's the relevant portion of my grub config:
I think 'iommu=1' is related to previous versions of Xen, and I'm not sure what 'iommu_inclusive_mapping=1' even does. I haven't verified that I get the same current functionality without these lines yet.
My 290x doesn't seem to support FLR (function level reset). I'm not sure if this is a requirement of all GPUs that will be passed to a VM or just Nvidia cards, but regardless I found this out by running sudo lspci -vv looking for my 290x and then finding the section starting with 'DevCap' (FLReset- is present). For what it's worth, all my devices that report on this functionality are listed as no support (FLReset-). First I need to ensure this is critical to successful PCI passthrouth. If so, then this makes me think I need to either 1) enable something in the BIOS or 2) change a kernel or module parameter.
Here is the output of of xl dmesg:
Code:
Xen 4.4.0 (XEN) Xen version 4.4.0 ([email protected]) (gcc (GCC) 4.9.0 20140521 (prerelease)) debug=n Sat May 31 15:52:14 PDT 2014 (XEN) Latest ChangeSet: (XEN) Bootloader: GRUB 2.02~beta2 (XEN) Command line: /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1 dom0_mem=16G xsave=1 (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds (XEN) Disc information: (XEN) Found 0 MBR signatures (XEN) Found 0 EDD information structures (XEN) Multiboot-e820 RAM map: (XEN) 0000000000000000 - 0000000000058000 (usable) (XEN) 0000000000058000 - 0000000000059000 (reserved) (XEN) 0000000000059000 - 000000000009f000 (usable) (XEN) 000000000009f000 - 00000000000a0000 (reserved) (XEN) 0000000000100000 - 0000000069e4c000 (usable) (XEN) 0000000069e4c000 - 0000000069e53000 (ACPI NVS) (XEN) 0000000069e53000 - 000000006ab54000 (usable) (XEN) 000000006ab54000 - 000000006b085000 (reserved) (XEN) 000000006b085000 - 000000007d185000 (usable) (XEN) 000000007d185000 - 000000007d216000 (reserved) (XEN) 000000007d216000 - 000000007d25d000 (usable) (XEN) 000000007d25d000 - 000000007d398000 (ACPI NVS) (XEN) 000000007d398000 - 000000007df7d000 (reserved) (XEN) 000000007df7d000 - 000000007dfff000 type 20 (XEN) 000000007dfff000 - 000000007e000000 (usable) (XEN) 000000007f000000 - 00000000bf200000 (reserved) (XEN) 00000000f0000000 - 00000000f8000000 (reserved) (XEN) 00000000fec00000 - 00000000fec01000 (reserved) (XEN) 00000000fed00000 - 00000000fed04000 (reserved) (XEN) 00000000fed1c000 - 00000000fed20000 (reserved) (XEN) 00000000fee00000 - 00000000fee01000 (reserved) (XEN) 00000000ff000000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 000000083fe00000 (usable) (XEN) ACPI: RSDP 000F0010, 0024 (r2 ALASKA) (XEN) ACPI: XSDT 7D365088, 0094 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FACP 7D376090, 010C (r5 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: DSDT 7D3651B8, 10ED6 (r2 ALASKA A M I 35 INTL 20120711) (XEN) ACPI: FACS 7D397F80, 0040 (XEN) ACPI: APIC 7D3761A0, 0092 (r3 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: FPDT 7D376238, 0044 (r1 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: SSDT 7D376280, 0539 (r1 PmRef Cpu0Ist 3000 INTL 20051117) (XEN) ACPI: SSDT 7D3767C0, 0B74 (r1 CpuRef CpuSsdt 3000 INTL 20051117) (XEN) ACPI: SSDT 7D377338, 01C7 (r1 PmRef LakeTiny 3000 INTL 20051117) (XEN) ACPI: MCFG 7D377500, 003C (r1 ALASKA A M I 1072009 MSFT 97) (XEN) ACPI: HPET 7D377540, 0038 (r1 ALASKA A M I 1072009 AMI. 5) (XEN) ACPI: SSDT 7D377578, 036D (r1 SataRe SataTabl 1000 INTL 20120711) (XEN) ACPI: SSDT 7D3778E8, 5B5E (r1 SaSsdt SaSsdt 3000 INTL 20120711) (XEN) ACPI: AAFT 7D37D448, 049F (r1 ALASKA OEMAAFT 1072009 MSFT 97) (XEN) ACPI: BGRT 7D37D8E8, 0038 (r0 ALASKA A M I 1072009 AMI 10013) (XEN) ACPI: DMAR 7D37D920, 00B8 (r1 INTEL BDW 1 INTL 1) (XEN) ACPI: SSDT 7D37D9D8, 0579 (r1 Intel_ IsctTabl 1000 INTL 20120711) (XEN) System RAM: 31690MB (32450764kB) (XEN) Domain heap initialised (XEN) ACPI: 32/64X FACS address mismatch in FADT - 7d397f80/0000000000000000, using 32 (XEN) Processor #0 7:12 APIC version 21 (XEN) Processor #2 7:12 APIC version 21 (XEN) Processor #4 7:12 APIC version 21 (XEN) Processor #6 7:12 APIC version 21 (XEN) Processor #1 7:12 APIC version 21 (XEN) Processor #3 7:12 APIC version 21 (XEN) Processor #5 7:12 APIC version 21 (XEN) Processor #7 7:12 APIC version 21 (XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 3598.990 MHz processor. (XEN) Initing memory sharing. (XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7 (XEN) spurious 8259A interrupt: IRQ7. (XEN) Intel VT-d iommu 0 supported page sizes: 4kB. (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB. (XEN) Intel VT-d Snoop Control not enabled. (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. (XEN) Intel VT-d Queued Invalidation enabled. (XEN) Intel VT-d Interrupt Remapping enabled. (XEN) Intel VT-d Shared EPT tables not enabled. (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) Platform timer is 14.318MHz HPET (XEN) Allocated console ring of 16 KiB. (XEN) VMX: Supported advanced features: (XEN) - APIC MMIO access virtualisation (XEN) - APIC TPR shadow (XEN) - Extended Page Tables (EPT) (XEN) - Virtual-Processor Identifiers (VPID) (XEN) - Virtual NMI (XEN) - MSR direct-access bitmap (XEN) - Unrestricted Guest (XEN) - VMCS shadowing (XEN) HVM: ASIDs enabled. (XEN) HVM: VMX enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) Brought up 8 CPUs (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f5b000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000820000000->0000000828000000 (4158762 pages to be allocated) (XEN) Init. ramdisk: 000000083f32a000->000000083fdff200 (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff81f5b000 (XEN) Init. ramdisk: ffffffff81f5b000->ffffffff82a30200 (XEN) Phys-Mach map: ffffffff82a31000->ffffffff84a31000 (XEN) Start info: ffffffff84a31000->ffffffff84a314b4 (XEN) Page tables: ffffffff84a32000->ffffffff84a5b000 (XEN) Boot stack: ffffffff84a5b000->ffffffff84a5c000 (XEN) TOTAL: ffffffff80000000->ffffffff84c00000 (XEN) ENTRY ADDRESS: ffffffff818ee1f0 (XEN) Dom0 has maximum 8 VCPUs (XEN) Scrubbing Free RAM: .....................................................................................................................................................done. (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Std. Loglevel: Errors and warnings (XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings) (XEN) Xen is relinquishing VGA console. (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 260kB init memory.
Code:
name = 'vwin' builder = 'hvm' memory = 8192 vcpus = 2 disk = [ '/home/joshua/vms/vwin.img,raw,xvda,w', 'file:/home/joshua/vms/win81.iso,hdc:cdrom,r' ] vif = [ 'bridge=xenbr0,ip=192.168.1.115' ] vnclisten = '0.0.0.0' vnc = 1 pci = [ '01:00.0', '01:00.1' ] pci_permissive = 1
Code:
Parsing config from /etc/xen/vwin.hvm libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0 libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1
Code:
This device cannot find enough free resources that it can use. (Code 12)
Code:
multiboot /xen-4.4.0.gz /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1 dom0_mem=16G xsave=1 module /vmlinuz-linux /vmlinuz-linux root=UUID=917153ed-f386-4987-8fd0-9e293f8d969a rw modprobe.blacklist=radeon i915 quiet rootflags=subvol=rootvol console=tty0 xen-pciback.passthrough=1 xen-pciback.hide=(01:00.0)(01:00.1)
Comment