Update
This knowledge base article will be used discuss the performance impacts of security issues related to speculative execution described by CVE-2018-3646 (L1 Terminal Fault - VMM) in modern-day processors as they apply to VMware.
Review the following Knowledge Base articles for a holistic view of VMware’s response to these issues:
Note: To aid in understanding the performance impact of Speculative Execution vulnerabilities to virtualization environments, VMware previously defined four mitigation categories in KB52245 and KB54951. Review these knowledge base articles for more detail if needed.
The Update History section of this article will be revised when there is a significant change to any of the related documentation. Click Subscribe to Article in the Actions box to be alerted when new information is added to this document and sign up at our Security-Announce mailing list to receive new and updated VMware Security Advisories.
VMware Security Advisories document remediation for security vulnerabilities that are reported in VMware products. Sign up on the right-hand side of this page to receive new and updated advisories in e-mail. How to enable VMWare Distributed Resource Scheduler (DRS) In this post, we’ll learn the steps to enable VMWare Distributed Resource Scheduler on vCenter Server 6.Vmware Distributed Resource Scheduler is used to automated load balance computing capacity to deliver optimized performance for hosts and virtual machines. Enable the ESXi Side-Channel-Aware Scheduler in ESXi 5.5, 6.0, 6.5, and 6.7 prior to 6.7u2. Notes: Enabling this option will result in the vSphere UI reporting only a single logical processor per physical core; halving the number of logical processors if Hyperthreading was previously enabled.
4/11/2019: VMware has published a white paper entitled Performance of vSphere 6.7 Scheduling Options which provides a detailed look into the performance differences between the original ESXi Side-Channel-Aware Scheduler and the ESXi Side-Channel-Aware Scheduler Version 2 which first shipped with ESXi 6.7u2.This knowledge base article will be used discuss the performance impacts of security issues related to speculative execution described by CVE-2018-3646 (L1 Terminal Fault - VMM) in modern-day processors as they apply to VMware.
Review the following Knowledge Base articles for a holistic view of VMware’s response to these issues:
- KB55636: VMware Overview of Speculative-Execution security issues in Intel processors: CVE-2018-3646 (L1 Terminal Fault - VMM), CVE-2018-3620 (L1 Terminal Fault - OS), and CVE-2018-3615 (L1 Terminal Fault - SGX, SMM)
- KB55806: VMware response to speculative-execution vulnerabilities in Intel processors: CVE-2018-3646 (“L1 Terminal Fault - VMM”)
Note: To aid in understanding the performance impact of Speculative Execution vulnerabilities to virtualization environments, VMware previously defined four mitigation categories in KB52245 and KB54951. Review these knowledge base articles for more detail if needed.
The Update History section of this article will be revised when there is a significant change to any of the related documentation. Click Subscribe to Article in the Actions box to be alerted when new information is added to this document and sign up at our Security-Announce mailing list to receive new and updated VMware Security Advisories.
by Sven Huisman
In 2018 the CPU hardware vulnerabilities Spectre, Meltdown and later L1 Terminal Fault (L1TF, also known as ForeShadow) made headlines. Hardware and software vendors released firmware updates and software patches to mitigate these vulnerabilities. Unfortunately, depending on the workload, these mitigations come with a performance impact. This GO-EUC research focusses on the impact on the performance of enabling the mitigations on the Windows server OS in an RDSH environment.
Spectre, Meltdown and L1 Terminal Fault
By now you should have heard about the hardware vulnerabilities that exist in modern processors and the exploits called Spectre, Meltdown and L1TF. You are probably also aware of the fact that hardware and software vendors released patches to mitigate against these vulnerabilities and that there might be a performance impact implementing and enabling these patches. If you don’t know about any of these vulnerabilities, start by reading Meltdownattack.com.
Protect your infrastructure
Vmware Esxi Side-channel-aware Scheduler
In most cases, after installing the necessary patches, you are protected against Spectre and Meltdown. On firmware-level, BIOS-level and Hypervisors, the mitigations are enabled by default. In Windows 10, the Spectre and Meltdown mitigations are enabled by default as well. In Windows server operating systems, it’s different. In the following table the default settings in Windows Operating systems are presented:
| OS | Spectre | Meltdown |
|---|---|---|
| Windows 10 | Enabled | Enabled |
| Windows Server 2016 or earlier | Disabled | Disabled |
| Windows Server 2019 | Disabled | Enabled |
For more information about Spectre and Meltdown on Windows Operating Systems, read the support article on the Microsoft website. This article describes how to enable or disable the mitigations in Windows.
To address the risk from L1TF you are required to take an extra step. In VMware vSphere the so called Side-Channel-Aware scheduler is available. On Hyper-V there is a similar feature called the Hyper-V Core Scheduler. Performance-wise, enabling these schedulers can have the same impact as disabling Hyperthreading.
Configuration and infrastructure
The tests for this research were performed on a Nutanix-lab environment. The node on which the tests were performed is part of a 4-node cluster. In a Nutanix cluster a controller VM (CVM) is present on each Nutanix node, but for these tests this CVM was powered off. The storage of the other nodes in the cluster was accessed by the test-node over NFS. For this test, a node with a dual socket Intel Xeon Gold 6130 Processor was used. This CPU is based on a Skylake architecture, containing 16 cores. That means a total of 32 physical CPU cores are present and with Hyperthreading enabled 64 logical CPU cores are available. VMware ESXi 6.7U1 was used during these tests. You can read more about this Nutanix-lab infrastructure here.
GO-EUC’s default workload was used (described here) with two modifications. The Remote Desktop Analyzer (RDA) data capture was disabled due to the overhead of RDA on RDSH and the PDF-printer was not disabled. SBC timer was not enabled during the tests.
RDSH VM configurations
In the test-host there are 32 physical cores available, 64 logical cores with hyperthreading. The ideal RDSH virtual machine setup in this configuration is 8 virtual machines with 8 vCPU each. Each VM is configured with 42GB of RAM. The installed operating system is Windows Server 2016 with Office 2016 and patches are included until February 2019.
Before performing the tests, all relevant patches were installed (with mitigations against Spectre, Meltdown and L1TF) on all levels in the infrastructure (BIOS, hardware firmware, hypervisor and guest OS). Because Spectre and Meltdown mitigations are disabled by default in Windows Server 2016, we set registry keys to enable Spectre and Meltdown mitigations when we tested the impact of Spectre and meltdown.The following three scenarios were tested:
- Windows server 2016 RDSH, with all relevant patches installed. Spectre and Meltdown are disabled by default. The Side-Channel Aware scheduler was not enabled on ESXi. This test is considered the baseline;
- The same Windows server 2016 image, but with Spectre and Meltdown enabled. The Side-Channel Aware scheduler was not enabled on ESXi;
- The same Windows server 2016 image with Spectre and Meltdown enabled and the Side-Channel Aware scheduler on VMware ESXi enabled as well.
The results
It is expected that Spectre, Meltdown and L1TF will have an impact on user density and user experience. Using Login VSI we can measure the impact by comparing the Login VSI VSImax and the Login VSI baseline. The VSImax is one of the best metrics to see if there is a capacity (user density) improvement. More information about the VSImax can be found here.
Higher is better
Enabling Spectre and Meltdown in a virtualized RDSH environment will decrease the VSImax with 9%. If the L1TF mitigation is also enabled, the VSImax will decrease with 50%! Enabling this mitigation (enabling the Side-Channel Aware scheduler on ESXi in this case) will result in not being able to leverage Hyperthreading, which partially explains this high impact.
Lower is better
The impact of enabling Spectre and Meltdown mitigations in the Windows server OS will cause the response times (Login VSI baseline) to go up with 9%. If you then enabled the Side-Channel Aware scheduler on ESXi as well, to mitigate the L1TF vulnerability, the response times will be back to almost the same level as before, only 2% difference from the baseline. This is probably because the VMs are now leveraging real CPU cores instead of hyperthreaded cores.The impact on the VSImax score and VSI baseline are most likely a result of a higher load on the CPU.
Lower is better
Lower is better
On average, the CPU utilization is 5% higher with Spectre and Meltdown mitigations enabled, and with the Side-Channel Aware scheduler enabled 17% higher. The trend is similar as the Login VSI results, but the impact on CPU utilization is lower than expected. Other aspects might influence the results of Login VSI as well, like CPU ready times.
Lower is better
Lower is better
Lower is better
Lower is better
As expected, there is no impact on reads enabling the mitigations. There is a small increase in writes when the average writes during the test are compared, but the chart of the Host writes/sec during the tests look very similar.
Lower is better
Lower is better
