Cette variable permet de diviser les objets en plusieurs parties et de les répartir sur plusieurs disques au sein du Cluster.
Exemple : StripeWidth=2, NumberOfFailuresToTolerate (FTT) = 1
Prenant en compte un Cluster de 3 nœuds et chaque nœud possède deux groupes de disques. Il y a donc en tout 6 groupes de disques.
Pour chaque groupe de disque il y a un SSD pour le “Cache Tiers” et 1 disques mécaniques pour le “Capacity Tiers”.
Cette configuration (StripeWidth=2) est effectivement juste car pour la première copie de la VM, les 2 “Stripes” seront partagés sur deux groupes de disques appartenant au même nœud et pour le Replica de la VM, les 2 “Stripes” seront répartis sur un groupes de disques des deux nœuds restants.
Pour une configuration (StripeWidth=3) vous aurez un Stripe sur chaque groupe de disque sur les 3 nœuds. Pour éviter en cvas de maintenance ou de défaillance de ce retrouver avec des “stripes” adjacents à d’autres “stripes” de la même VM, il faut considérer l’ajout d’un quatrième nœud pour le Cluster.
Si pour la même configuration vous configurez le StripeWidth=4, les “Stripes” d’un même objet pourrons se retrouver sur plusieurs nœuds et donc la variable NumberOfFailuresToTolerate (FTT) = 1, ne pourra plus être respectée car les datas d’une même VM pourront être à cheval sur plusieurs nœuds.
VMware Virtual SAN 6.1, shipping with vSphere 6.0 Update 1, introduced a new feature called VMware Virtual SAN Stretched Cluster. Virtual SAN Stretched Cluster is a specific configuration implemented in environments where disaster/downtime avoidance is a key requirement. This guide was developed to provide additional insight and information for installation, configuration and operation of a Virtual SAN Stretched Cluster infrastructure in conjunction with VMware vSphere. This guide will explain how vSphere handles specific failure scenarios and discuss various esign considerations and operational procedures.
Virtual SAN Stretched Clusters with Witness Host refers to a deployment where a user sets up a Virtual SAN cluster with 2 active/active sites with an identical number of ESXi hosts distributed evenly between the two sites. The sites are connected via a high bandwidth/low latency link.
The third site hosting the Virtual SAN Witness Host is connected to both of the active/active data-sites. This connectivity can be via low andwidth/high latency links.
Each site is configured as a Virtual SAN Fault Domain. The nomenclature used to describe a Virtual SAN Stretched Cluster configuration is X+Y+Z, where X is the number of ESXi hosts at data site A, Y is the number of ESXi hosts at data site B, and Z is the number of witness hosts at site C. Data sites are where virtual machines are deployed. The minimum supported configuration is 1+1+1(3 nodes). The maximum configuration is 15+15+1
(31 nodes). In Virtual SAN Stretched Clusters, there is only one witness host in any configuration.
A virtual machine deployed on a Virtual SAN Stretched Cluster will have one copy of its data on site A, a second copy of its data on site B and any witness components placed on the witness host in site C. This configuration is achieved through fault domains alongside hosts and VM groups, and affinity rules. In the event of a complete site failure, there will be a full copy of the virtual machine data as well as greater than 50% of the components available. This will allow the virtual machine to remain available on the Virtual SAN datastore. If the virtual machine needs to be restarted on the other site, vSphere HA will handle this task.
Virtual SAN Stretched Cluster configurations require vSphere 6.0 Update 1 (U1) or greater. This implies both vCenter Server 6.0 U 1 and ESXi 6.0 U1. This version of vSphere includes Virtual SAN version 6.1. This is the minimum version required for Virtual SAN Stretched Cluster support.
Virtual SAN version 6.1 introduced features including both All-Flash and Stretched Cluster functionality. There are no limitations on the edition of vSphere used for Virtual SAN. However, for Virtual SAN Stretched Cluster functionality, vSphe re DRS is very desi rable . DRS will provide initial placement assistance, and will also automatically migrate virtual machines to their co rrect site in accordance to Host/VM affinity rules. It can also help will locating virtual machines to their co rrect site when a site recovers after a failure. Otherwise the administrator will have to manually carry out these tasks. Note that DRS is only available in Enterprise edition and higher of vSphere.
Virtual SAN Stretched Cluster is supported on both hybrid configurations (hosts with local storage comprised of both magnetic disks for capacity and flash devices for cache) and all-flash configurations (hosts with local storage made up of flash devices for capacity and flash devices for cache).
VMware supports Virtual SAN Stretched Cluster with the v2 on-disk format only. The v1 on-disk format is based on VMFS and is the original on-disk format used for Virtual SAN. The v2 on-disk format is the version which comes by default with Virtual SAN version 6.x. Customers that upgraded from the original Virtual SAN 5.5 to Virtual SAN 6.0 may not have upgraded the on-disk format for v1 to v2, and are thus still using v1. VMware ecommends upgrading the on-disk format to v2 for improved performance and scalability, as well as stretched cluster support. In Virtual SAN 6.2 clusters, the v3 on-disk format allows for additional features, discussed later, specific to 6.2.
Both physical ESXi hosts and virtual ESXi hosts (nested ESXi) are supportedfor the witness host. VMware provides a Witness Appliance for those customers who wish to use the ESXi VM. A witness host/VM cannot be shared between multiple Virtual SAN Stretched Clusters.
Features Supported on VSAN but not VSAN Stretched
The following are a list of products and features support on Virtual SAN but not on a stretched cluster implementation of Virtual SAN.
Features Supported on vMSC but not VSAN Stretched
The following are a list of products and features support on vSphere Metro Storage Cluster (vMSC) but not on a stretched cluster implementation of Virtual SAN.
A common question is how stretched cluster differs from Fault Domains, which is a Virtual SAN feature that was introduced with Virtual SAN version 6.0. Fault domains enable what might be termed “rack awareness” where the components of virtual machines could be distributed amongst multiple hosts in multiple racks, and should a rack failure event occur, the virtual machine would continue to be available. However, these racks would typically be hosted in the same data center, and if there was a data center wide event, fault domains would not be able to assist with virtual machines availability.
Stretched clusters essentially build on what fault domains did, and now provide what might be termed “data center awareness”. Virtual SAN Stretched Clusters can now provide availability for virtual machines even if a data center suffers a catastrophic outage.
The witness host is a dedicated ESXi host (or appliance) whose purpose is to host the witness component of virtual machines objects. The witness must have connection to both the master Virtual SAN node and the backup Virtual SAN node to join the cluster. In steady state operations, the master node resides in the “preferred site”; the backup node resides in the “secondary site”. Unless the witness host connects to both the master and the backup nodes, it will not join the Virtual SAN cluster.
In traditional Virtual SAN clusters, a virtual machine’s read operations are distributed across all replica copies of the data in the cluster. In the case of a policy setting of NumberOfFailuresToTolerate=1, which results in two copies of the data, 50% of the reads will come from replica1 and 50% will come from replica2. In the case of a policy setting of Number Of Failures To Tolerate=2 in non-stretched Virtual SAN clusters, results in three copies of the data, 33% of the reads will come from replica1, 33% of the reads will come from replica2 and 33% will come from replica3.
In a Virtual SAN Stretched Cluster, we wish to avoid increased latency caused by reading across the inter-site link. To insure that 100% of reads, occur in the site the VM resides on, the read locality mechanism was introduced. Read locality overrides the NumberOfFailuresToTolerate=1 policy’s behavior to distribute reads across the two data sites.
DOM, the Distributed Object Manager in Virtual SAN, takes care of this. DOM is responsible for the creation of virtual machine storage objects in the Virtual SAN cluster. It is also responsible for providing distributed data access paths to these objects. There is a single DOM owner per object. There are 3 roles within DOM; Client, Owner and Component Manager. The DOM Owner coordinates access to the object, including reads, locking and object configuration and reconfiguration. All objects changes and writes also go through the owner. The DOM owner of an object will now take into account which fault domain the owner runs in a Virtual SAN Stretched Cluster configuration, and will read from the replica that is in the same domain.
There is now another consideration with this read locality. One must avoid unnecessary vMotion of the virtual machine between sites. Since the read cache blocks are stored on one site, if the VM moves around freely and ends up on the remote site, the cache will be cold on that site after the move. Now there will be sub-optimal performance until the cache is warm again. To avoid this situation, soft affinity rules are used to keep the VM local to the same site/fault domain where possible. The steps to configure such rules will be shown in detail in the vSphere DRS section of this guide.
Virtual SAN 6.2 introduced Client Cache, a mechanism that allocates 0.4% of host memory, up to 1GB, as an additional read cache tier. Virtual machines leverage the Client Cache of the host they are running on. Client Cache is not associated with Stretched Cluster read locality, and runs ndependently.
A Virtual SAN Stretched Cluster configuration can be created and managed by a single instance of VMware vCenter Server. Both the Windows version and the Virtual Appliance version (Linux) are supported for configuration and management of a Virtual SAN Stretched Cluster.
In a Virtual SAN Stretched Cluster, the witness components are only ever placed on the witness host. Either a physical ESXi host or a special witness appliance provided by VMware, can be used as the witness host.
If a witness appliance is used for the witness host, it will not consume any of the customer’s vSphere licenses. A physical ESXi host that is used as a witness host will need to be licensed accordingly, as this can still be used to provision virtual machines should a customer choose to do so.
It is important that witness host is not added to the VSAN cluster. The witness host is selected during the creation of a Virtual SAN Stretched Cluster.
The witness appliance will have a unique identifier in the vSphere web client UI to assist with identifying that a host is in fact a witness appliance (ESXi in a VM). It is shown as a “blue” host, as highlighted below:
Note this is only visible when the appliance ESXi witness is deployed. If a physical host is used as the witness, then it does not change its appearance in the web client. A witness host is dedicated for each stretchd cluster.
When Virtual SAN is deployed in a stretched cluster across multiple sites using fault domains, there are certain networking requirements that must be adhered to.
Both Layer 2 (same subnet) and Layer 3 (routed) configurations are used in a recommended Virtual SAN Stretched Cluster deployment.
Note: A common question is whether L2 for Virtual SAN traffic across all sites is supported. There are some considerations with the use of a stretched
L2 domain between the data sites and the witness site, and these are discussed in further detail in the design considerations section of this guide. Another common question is whether L3 for VSAN traffic across all sites is supported. While this can work, it is not the VMware recommended network topology for Virtual SAN Stretched Clusters at this time.
Virtual SAN traffic between data sites is multicast. Witness traffic between a data site and the witness site is unicast.
For VMware Virtual SAN Stretched Cluste rs, geographical distances are not a support concern. The key requirement is the actual latency numbers between sites.
Data site to data site network refers to the communication between non-witness sites, in other words, sites that run virtual machines and hold virtual machine data. Latency or RTT (Round Trip Time) between sites hosting virtual machine objects should not be greater than 5msec (< 2.5msec one-way).
Bandwidth between sites hosting virtual machine objects will be workload dependent. For most workloads, VMware recommends a minimum of 10Gbps or greater bandwidth between sites. In use cases such as 2 Node configurations for Remote Office/Branch Office deployments, dedicated 1Gbps bandwidth can be sufficient with less than 10 Virtual Machines.
Please refer to the Design Considerations section of this guide for further details on how to determine bandwidth requirements.
This refers to the communication between non-witness sites and the witness site.
In most Virtual SAN Stretched luster configurations, latency or RTT (Round Trip Time) between sites hosting VM objects and the witness nodes should not
be greater than 200msec (100msec one-way).
In typical 2 Node configurations, such as Remote Office/Branch Office deployments, this latency or RTT is supported up to 500msec (250msec one-way).
The latency to the witness is dependent on the number of objects in the cluster. VMware recommends that on Virtual SAN Stretched lusterconfigurations up to 10+10+1, a latency of less than or equal to 200 milliseconds is acceptable, although if possible, a latency of less than or equal to 100 milliseconds is preferred. For configurations that are greater than 10+10+1, VMware recommends a latency of less than or equal to 100 milliseconds is required.
Bandwidth between sites hosting VM objects and the witness nodes are dependent on the number of objects residing on Virtual SAN. It is important to size data site to witness bandwidth appropriately for bo th availability and growth. A standard rule of thumb is 2Mbps for every 1000 objects on Virtual SAN.
Please refer to the Design Considerations section of this guide for further details on how to determine bandwidth requirements.
It is important to maintain a consistent MTU size between data nodes and the witness in a Stretched Cluster configuration. Ensuring that each VMkernel interface designated for Virtual SAN traffic, is set to the same MTU size will prevent traffic fragmentation. The Virtual SAN Health Check checks for a uniform MTU size across the Virtual SAN data network, and reports on any inconsistencies.
The maximum number of virtual machines per ESXi host is unaffected by the Virtual SAN Stretched Cluster configuration. The maximum is the same as for normal VSAN deployments.
VMware recommends that customers should run their hosts at 50% of maximum number of virtual machines supported in a standard Virtual SAN cluster to accommodate a full site failure. In the event of full site failures, the virtual machines on the failed site can be restarted on the hosts in the surviving site.
The minimum number of hosts in a Virtual SAN Stretched Cluster is 3. In such a configuration, site 1 will contain a single ESXi host, site 2 will contain a single ESXi host and then there is a witness host at the third site, the witness site. The nomenclature for such a configuration is 1+1+1. This is commonly referred to as a 2 Node configuration.
The maximum number of hosts in a Virtual SAN Stretched Cluster is 31. Site 1 contains ESXi 15 hosts, site 2 contains 15 ESXi hosts, and the witness host on the third site makes 31. This is referred to as a 15+15+1 configuration.
There is a maximum of 1 witness host per Virtual SAN Stretched Cluster. The witness host requirements are discussed in the design considerations section of this guide. VMware provides a fully supported witness virtual appliance, in Open Virtual Appliance (OVA) format, for customers who do not wish to dedicate a physical ESXi host as the witness. This OVA is essentially a pre-licensed ESXi host running in a virtual machine, and can be deployed on a physical ESXi host on the third site.
Because Virtual SAN Stretched Cluster configurations effectively have 3 fault domains, the Number Of Failures To Tolerate (FTT) policy setting, has a maximum of 1 for objects. Virtual SAN cannot comply with FTT values that are greater than 1 in a stretched cluster configuration.
Other policy settings are not impacted by deploying VSAN in a stretched cluster configuration and can be used as per a non-stretched VSAN cluster.
Fault domains play an important role in Virtual SAN Stretched Cluster. Similar to the Number Of Failures To Tolerate (FTT) policy setting discussed previously, the maximum number of fault domains in a Virtual SAN Stretched Cluster is 3. The first FD is the “preferred” data site, the second FD is the “secondary” data site and the third FD is the witness host site.
Make your job easier by simplifying storage provisioning and management for vSphere. Deploy storage with just a few mouse clicks from the vSphere Web Client and enjoy native integration with the VMware stack. Virtual machine-centric storage policies automate storage services levels on a per-VM basis.
Customers of all industries and sizes trust Virtual SAN to run their business-critical workloads, from key business applications to thousands of virtual desktops. Virtual SAN ensures that data is never lost if a disk, host, network or rack fails and can even tolerate entire site failures with synchronous replication and stretched clusters.
Deploy on inexpensive industry-standard server components to remove large, upfront investments. Eliminate siloed, purpose-built hardware and automate management of storage service levels through VM-centric policies. Further improve TCO with storage efficiency features like deduplication and enhanced automation capabilities.
Built on an optimized I/O data path in the hypervisor and designed for flash speeds, Virtual SAN delivers much better performance than a virtual appliance or external device. Experience up to 100,000 IOPs per host with all-flash and scale up to 64 hosts per cluster—a perfect match for virtual desktops, remote IT and business critical applications.
Trust your mission-critical applications with the only hypervisor-embedded storage solution. Virtual SAN delivers all-flash performance with up to 100,000 IOPs per host, support for vSphere availability technologies like High Availability (HA), asynchronous replication, stretched cluster capabilities, and storage efficiency features for all-flash including deduplication, compression and erasure coding.
Virtual SAN delivers predictable, elastic and non-disruptive scaling of storage and compute resources, eliminating costly forklift upgrades. Every Virtual SAN cluster can scale out one node at a time or scale up by adding capacity to existing hosts and is capable of achieving over 8 PB of raw storage capacity.
Virtual SAN is the only hyper-converged storage solution delivered directly from the hypervisor, resulting in a flash-optimized architecture that delivers as much as 100,000 IOPS per all-flash node.
Leverage the shortest, most efficient I/O data path for optimized performance with the least impact on CPU and memory resources.
Deploy an all-flash architecture with flash-based caching and SSD data persistence to provide up to 100,000 IOPS per host with consistent, low latency.
Improve virtual machine performance and minimize storage latencies by caching and accelerating read/write disk I/O traffic through enterprise-grade, server-side flash cache.
Automatically rebuild and rebalance storage to align with the storage service levels assigned to each virtual machine in the cluster.
Automatically limits and monitors the IOPS consumed by specific virtual machines, eliminating noisy neighbor issues.
Virtual SAN enables the most cost-efficient all-flash performance by delivering up to 10x greater storage utilization through data reduction technologies to optimize data footprint.
Optimizes all-flash storage capacity with as much as 7x data reduction while having minimal impact on server CPU and memory resources.
Increases usable storage capacity by up to 2x with the same high levels of data resiliency. Tolerate one or two failures with single parity or double parity protection.
Virtual SAN delivers enterprise availability for the most demanding business critical applications, capable of delivering 99.999% uptime and beyond with built-in and tunable failure tolerance settings.
Maximize resiliency with built-in distributed RAID, cache mirroring, and per-VM control by specifying how many host, network, disk, or rack failures to tolerate.
Enable enterprise-level availability that ensures no data loss and near zero downtime even in the event of an entire site failure with synchronous replication between two data centers.
Leverage core vSphere features such as High Availability (HA) and Distributed Resource Scheduler (DRS) along with vSphere snapshots and VMware Site Recovery Manage.
vSphere Replication for Virtual SAN provides asynchronous VM replication with RPOs of up to five minutes.
Virtual SAN provides the simplicity of managing storage along with compute and networking in a single, tightly integrated interface–the vSphere Web Client.
Eliminate the need for training on specialized storage interfaces and tools by using vSphere Web Client-based management to provision storage in two clicks.
Deploy storage policies with a single click to automatically provision storage resources–no LUNs or RAID configurations necessary.
Perform hardware, firmware, and driver compatibility checks directly from the vSphere Web Client, along with real-time diagnostics and reporting on performance and storage capacity.
Virtual SAN is radically simple, enterprise-class software-defined storage powering VMware hyper-converged infrastructure. Get started by learning about Virtual SAN at your own pace or learn with us. We offer a variety of ways to experience the benefits of hyper-converged infrastructure (HCI) and Virtual SAN, from live webinars to self-paced on-demand courses.
Virtual SAN has specific requirements for hardware (hosts, CPU, memory, number & type of storage devices), software (version of vCenter, vSphere configuration, ESXi host version & configuration), and networking.
In traditional configurations, Virtual SAN requires 3 hosts, with at least one disk group each, configured in a vSphere Cluster with Virtual SAN enabled, with a VMkernel port configured for Virtual SAN traffic that has connectivity to other hosts in the cluster. Note* Multicast traffic is required.
More detailed information for the different versions of Virtual SAN can be found in the relative KB article for each version of Virtual SAN.
Hosts that are certified to run VMware vSphere along with components that are certified to run Virtual SAN. Components certified for Virtual SAN are tested by the VMware Storage and Availability Business Unit.
Virtual SAN can be installed on hardware in a bring your own component, Ready Node, or Engineered Appliance offering.
Bring Your Own
The VMware vSphere Hardware Compatibility List details which hosts and CPU configurations are supported with ESXi.
The vSphere HCL can be found here: http://vmwa.re/hcl
When combined with the vSphere HCL, the Virtual SAN Compatibility Guide provides a list of hardware certified to run Virtual SAN.
The Virtual SAN Component HCL can be found here: http://vmwa.re/vsanhclc/
Ready Nodes are a pre-validated hardware options from various OEMs.
Ready Nodes can be found here: http://vmwa.re/vsanhcl/
The Ready Node selector can be found here: http://vsanreadynode.vmware.com/
Warning: Using uncertified hardware may lead to performance issues and/or data loss. The reason for this is that the behavior of uncertified hardware cannot be predicted. VMware cannot provide support for environments running on uncertified hardware.
Cluster requirements include
What are the memory requirements of Virtual SAN 6?
Memory requirements are determined by the number of disk groups and devices that are managed by ESXi.
Hosts should contain at least 32GB of RAM to accommodate the maximum number of disk groups and devices.
To calculate Virtual SAN 6.0 memory consumption , use this equation:
BaseConsumption + (NumDiskGroups x ( DiskGroupBaseConsumption + (SSDMemOverheadPerGB x SSDSize)))
KB Article 2113954 provides scenarios to better illustrate Virtual SAN memory requirements.
Processors that are approved on the vSphere Compatibility Guide are approved for use with Virtual SAN.
The software requirements for Virtual SAN 6 are:
The networking requirements for Virtual SAN 6 are:
Note: When running multiple Virtual SAN clusters, it is required to have isolated Virtual SAN networks.
This isolation can be achieved through the use of:
For more information, see Networking Requirements for Virtual SAN section in the VMware Virtual SAN 6.0 Documentation.
The storage requirements are:
*Up to 600GB
Virtual SAN licensing varies by consumption type, version and edition
Virtual SAN may be used with any edition of vSphere. It is licensed by method and edition.
VMware Virtual SAN is radically simple, enterprise-class storage for hyper-converged infrastructure (HCI). Uniquely embedded in the VMware vSphere hypervisor, Virtual SAN delivers flash-optimized, high-performance storage. It leverages commodity x86 server components to drastically lower TCO by up to 50% and deliver all-flash solutions for as low as half the price of competitive hybrid HCI systems. Seamless integration with vSphere and the entire VMware stack makes it the simplest storage platform for business-critical workloads, virtual desktop infrastructure (VDI), remote office IT, disaster recovery, and DevOps infrastructures. Customers of all industries and sizes trust Virtual SAN to run their most important applications.
Use cases for Virtual SAN include mission-critical applications, test and development, remote and branch offices, disaster recovery sites, and virtual desktop infrastructure (VDI). Virtual SAN supports a number of configuration options such as 2-node clusters for small implementations at remote offices to clusters as large as 64 nodes delivering over 6 million IOPS. Stretched clusters can also be configured to provide resiliency against site failure and workload migration with no downtime for disaster avoidance. VMware provides the market-leading HCI platform powered by vCenter Server, vSphere and Virtual SAN.
New in vSphere 6.5 and Virtual SAN 6.5 is the ability to separate witness network traffic from the network that connects the two physical hosts in a two-node cluster. This configuration option reduces complexity by eliminating the need to create and maintain static routes on the physical hosts. Security is also improved as the data network (between physical hosts) is completely separated from the WAN for witness traffic.
A Virtual SAN Witness is a virtual appliance that is utilized in stretched clusters and two-node configurations. The witness acts as a “tie-breaker” in certain events such as a network partition between hosts in a 2-node cluster. The witness virtual appliance stores metadata such as witness components. It does not store data components such as VM Home and virtual disk (VMDK) files.
Two-node VSAN cluster configurations are commonly used for branch offices and remote locations running a small number of virtual machines. A witness virtual appliance must not run on either of the two physical nodes in the cluster as this would defeat the purpose of the witness. The recommended location for the witness virtual appliance(s) is a primary data center connected to the remote locations by a WAN. Having the witness virtual appliances centrally located facilitates ease of management.
Physical Hosts in a two-node cluster can be connected using crossover cables. Organizations with existing 100Mbps or 1Gbps networking hardware can continue to utilize this equipment and enable 10Gbps connections for Virtual SAN and vMotion. All-flash Virtual SAN configurations (including two-node) require 10Gbps connectivity and vMotion performance is enhanced using faster connections between hosts. Connection reliability between the physical hosts is also improved with direct connections.
As with any storage fabric, redundant connections are recommended. In addition to providing resiliency against a NIC or cable failure, two connections between physical hosts enables administrators to separate Virtual SAN from vMotion traffic. It is possible for vMotion to saturate a 10Gbps link when migrating multiple virtual machine simultaneously, e.g., when a host is put into maintenance mode. This scenario has the potential to impact Virtual SAN performance while the migrations are taking place. The figure below shows one possible configuration that would prevent Virtual SAN and vMotion from using the same connection unless there is a NIC or cable failure.
Block storage can be provided to physical workloads using the iSCSI protocol. The Virtual SAN iSCSI target service provides flexibility and potentially avoids expenditure on purpose-built, external storage arrays. In addition to capital cost savings, the simplicity of Virtual SAN lowers operational costs.
The Virtual SAN iSCSI target service is enabled with just a few mouse clicks. CHAP and Mutual CHAP authentication is supported. Virtual SAN objects that serve as iSCSI targets are managed with storage policies just like virtual machine objects.
After the iSCSI target service is enabled, iSCSI targets and LUNs can be created. The screen shot below shows target and LUN configuration options.
The last step is adding initiator names or an initiator group, which controls access to the target, as shown here.
In nearly all cases, it is best to run workloads in virtual machines to take full advantage of Virtual SAN’s simplicity, performance, and reliability. However, for those use cases that truly need block storage, it is now possible to utilize Virtual SAN iSCSI Target Service.
New application architecture and development methods have emerged that are designed to run in today’s mobile-cloud era. For example, “DevOps” is a term that describes how these next-generation applications are developed and operated. “Container” technologies such as Docker and Kubernetes are a couple of the many solutions that have emerged as options for deploying and orchestrating these applications. VMware is embracing these new application types with a number products and solutions. Here are a few examples:
Photon OS – a minimal Linux container host optimized to run on VMware platforms.
Lightwave – an open source project comprised of enterprise-grade identity and access management services.
Photon Controller – a distributed, multi-tenant ESXi host controller optimized for containers.
Cloud native applications naturally require persistent storage just the same as traditional applications. Virtual SAN for Photon Controller enables the use of a Virtual SAN cluster in cloud native application environments managed by Photon Controller. Virtual SAN for Photon Controller includes developer-friendly APIs for storage provisioning and consumption. APIs and a graphical user interface (GUI) geared toward IT staff are also included for management and operations.
vSphere Integrated Containers Engine is a container runtime for vSphere, allowing developers familiar with Docker to develop in containers and deploy them alongside traditional virtual machine workloads on vSphere clusters. vSphere Integrated Containers Engine enables these workloads to be managed through the vSphere GUI in a way familiar to vSphere admins. Availability and performance features in vSphere and Virtual SAN can be utilized by vSphere Integrated Containers Engine just the same as traditional virtual machine environments.
Docker Volume Driver enables vSphere users to create and manage Docker container data volumes on vSphere storage technologies such as VMFS, NFS, and Virtual SAN. This driver makes it very simple to use containers with vSphere storage and provides the following key benefits:
Virtual SAN along with the solutions above provides an ideal storage platform for developing, deploying, and managing cloud native applications.
VMware’s PowerCLI implementation is one of the most widely adopted extensions to the framework. It features a plethora of functions that abstract the vSphere API down to simple and powerful cmdlets including a number of cmdlets for Virtual SAN. This makes it easy to automate a number of actions from simply enabling Virtual SAN to deployment and configuration of a Virtual SAN stretched cluster. Here are a few examples of what can be accomplished with Virtual SAN and PowerCLI:
With each new release of vSphere PowerCLI and Virtual SAN APIs, the functionality becomes more robust. Cmdlets have been added and improved for these and many other operations:
vSphere administrators and DevOps shops can utilize these new cmdlets to lower costs by enforcing standards, streamlining operations, and enabling automation for vSphere and Virtual SAN environments.
Disk drives have been using a native 512-byte sector size. Due to increasing demands for larger capacities, the storage industry introduced new formats that use 4KB physical sectors. These are commonly referred to as “4K native” drives or simply “4Kn” drives. Some 4Kn devices include firmware that emulates 512 byte (logical) sectors while the underlying (physical) sectors are 4K. These devices are referred to as “512B emulation” or “512e” drives.
vSphere 6.5 and Virtual SAN 6.5 support the use of 512e drives. The latest information regarding support for these new drive types can be found in this VMware Knowledge Base Article: Support statement for 512e and 4K Native drives for VMware vSphere and VSAN (2091600)
Virtual SAN clusters consist of any number of physical server hosts from two to 64. Each host contains flash devices (all-flash configuration) or a combination of magnetic disks and flash devices (hybrid configuration) that contribute cache and capacity to the Virtual SAN distributed datastore. Each host has one to five disk groups. Each disk group contains one cache device and one to seven capacity devices.
For all-flash configurations, the flash device(s) in the cache tier are used for write caching only as read performance from the capacity flash devices is more than sufficient. Two grades of flash devices are commonly used in an all-flash Virtual SAN configuration: Lower capacity, higher endurance devices for the cache layer and more cost effective, higher capacity, lower endurance devices for the capacity layer. Writes are performed at the cache layer and then de-staged to the capacity layer, only as needed. This helps extend the usable life of the lower endurance flash devices in the capacity layer.
In a hybrid configuration, one flash device and one or more magnetic drives are configured as a disk group. A disk group can have up to seven magnetic drives for capacity. One or more disk groups are utilized in a vSphere host depending on the number of flash devices and magnetic drives contained in the host. Flash devices serve as read and write cache for the Virtual SAN datastore while magnetic drives make up the capacity of the datastore. By default, Virtual SAN will use 70% of the flash capacity as read cache and 30% as write cache.
Virtual SAN Ready Nodes are typically the easiest, most flexible approach when considering deployment methods. Virtual SAN Ready Nodes are x86 servers, available from all of the leading server vendors that have been pre-configured, tested and certified for Virtual SAN. Each Ready Node is optimally configured for Virtual SAN with the required amount of CPU, memory, network, I/O controllers and storage devices.
Turn-key appliances such as Dell EMC VxRail provide a fully-integrated, preconfigured, and pre-tested VMware hyper-converged solution for a variety of applications and workloads. Simple deployment enables customers to be up and running in as little as 15 minutes.
Custom configurations using jointly validated components from a number of OEM vendors is also an option. The Virtual SAN Hardware Quick Reference Guide provides some sample server configurations as directional guidance and all components should be validated using the VMware Compatibility Guide for Virtual SAN.
All-flash Virtual SAN configurations require 10Gb network connectivity. 1Gb connections are supported for hybrid configurations although 10Gb is recommended. Currently, Virtual SAN requires the use of multicast on the network. Multicast communication is used for host discovery and to optimize network bandwidth consumption for the metadata updates. This eliminates the computing resource and network bandwidth penalties that unicast imposes in order to send identical data to multiple recipients.
Support for using crossover cables in a 2-node cluster is new in Virtual SAN 6.5. Utilizing crossover cables eliminates the need to procure and manage a 10Gb network switch for these hosts, which lowers costs – especially in scenarios such as remote office deployments. This configuration also improves availability and provides complete separation of Virtual SAN witness traffic from data traffic.
For more information on Virtual SAN network requirements and configuration recommendations, see the Virtual SAN Network Design Guide.
Storage in a hyper-converged infrastructure (HCI) requires compute resources that have been traditionally offloaded to dedicated storage arrays. Most HCI solutions require the deployment of storage virtual appliances to some or all of the hosts in the cluster to provide storage services to each host. These virtual appliances typically require CPU and/or memory reservations to avoid resource contention, which can result in performance degradation. Running a virtual appliance on every host in the cluster reduces the overall amount of compute resources available to run regular virtual machine workloads. Consolidation ratios will likely be lower and total cost of ownership rises when these storage virtual appliances are present and competing for the same resources as regular virtual machine workloads.
Storage virtual appliances can also introduce additional latency, which negatively affects performance. This is due to the number of steps required to handle and replicate write operations as shown in the figure below.
Virtual SAN does not require the deployment of storage virtual appliances or the installation of a vSphere Installation Bundle (VIB) on every host in the cluster. Virtual SAN is embedded in the vSphere kernel and typically consumes less than 10% of the compute resources on each host. Virtual SAN does not compete with other virtual machines for resources and the IO path is shorter.
A shorter IO path and the absence of resource-intensive storage virtual appliances enables Virtual SAN to provide extreme performance with minimal overhead. Higher consolidation ratios translate into lower total costs of ownership.
Virtual SAN is built into vSphere. Virtual SAN is enabled with just a few mouse clicks. There is no requirement to install additional software and/or deploy virtual storage appliances to every host in the cluster. Simply click the Enable Virtual SAN checkbox to start the process. Deduplication and compression can also be enabled at that time.
The next step is claiming local storage devices in each host for the Virtual SAN cache and capacity tiers. One or more disk groups are created in each host. Each disk group contains one cache device (flash) and one or more capacity devices (flash or magnetic). Virtual SAN pools these local storage device together to create a pool of shared storage.
The process of enabling Virtual SAN takes only a better of minutes. This is a tribute to the simplicity of Virtual SAN – especially when you compare it to other enterprise-class hyper-converged and traditional storage systems, which typically take much longer to set up.
Virtual SAN 6.2 features a comprehensive health service that actively tests and monitors a number of items such as hardware compatibility, network connectivity, cluster health, and capacity consumption. The health service is enabled by default and configured to check the health of the Virtual SAN environment every 60 minutes.
The Health service is quite thorough in the number of tests it performs. As an example, proper network configuration is essential to a healthy Virtual SAN cluster and there are 11 tests in the“Network” section of the Virtual SAN Health user interface.
If an issue is detected, a warning is visible in the Virtual SAN user interface. Clicking on the warning provides more details about the issue. For example, a controller driver that is not on the hardware compatibility list (HCL) will trigger a warning. In addition to providing details about the warning, Virtual SAN Health also has an“Ask VMware” button, which brings up the relevant VMware Knowledge Base article.
vSphere and Virtual SAN support a wide variety of hardware configurations. The list ofhardware components and corresponding drivers that are supported with Virtual SAN can be found in the VMware Compatibility Guide. It is very important to use only hardware, firmware, and drivers found in this guide to ensure stability and performance. The list of certified hardware, firmware, and drivers is contained in a hardware compatibility list (HCL) database. Virtual SAN makes it easy to update this information, for use by the Health Service tests. If the environment has Internet connectivity, updates can be obtained directly from VMware. Otherwise, HCL updates can be downloaded to enable offline updates.
If an issue does arise that requires the assistance of VMware Support, it is easy to upload support bundles to help expedite the troubleshooting process. Clicking the “Upload Support Bundles to Service Request…” button enables an administrator to enter an existing support request (SR) number and upload the necessary logs with just a few mouse clicks.
Figure x. Virtual SAN HCL Database and Support Assistant
Virtual SAN proactive tests enable administrators verify Virtual SAN configuration, stability, and performance to minimize risk and confirm that the datastore is ready for production use. Three proactive tests are available:
VM creation test
Multicast performance test
Storage performance test.
The VM creation test creates and deletes a small virtual machine on each host confirming basic functionality. The multicast test verifies multicast is working properly on each host and performance meets Virtual SAN requirements. The following figure shows the results of a VM creation test.
The storage performance test is used to check the stability of the Virtual SAN cluster under heavy I/O load. There are a number of workload profiles that can be selected for this test as shown below. Keep in mind the storage performance test can affect other workloads and tasks. This test is intended to run before production virtual machine workloads are provisioned on Virtual SAN.
Capacity overviews are available in the Virtual SAN user interface making it easy for administrators to see used and free space at a glance. Deduplication and compression information is also displayed.
Information is also available showing how much capacity various object types are consuming. Note that percentages are of used capacity, not of total capacity.
This list provides more details on the object types in the Used Capacity Breakdown chart:
Virtual disks: Virtual disk consumption before deduplication and compression
VM home objects: VM home object consumption before deduplication and compression
Swap objects: Capacity utilized by virtual machine swap files
Performance management objects: When the Virtual SAN performance service is enabled, this is the amount of capacity used to store the performance data
File system overhead: Capacity required by the Virtual SAN file system metadata
Deduplication and compression overhead: deduplication and compression metadata, such as hash, translation, and allocation maps
Checksum overhead: Capacity used to store checksum information
Other: Virtual machine templates, unregistered virtual machines, ISO files, and so on that are consuming Virtual SAN capacity
A healthy Virtual SAN environment is one that is performing well. Virtual SAN includes a number of graphs and data points that provide performance information at the cluster, host, virtual machine, and virtual disk levels. Time Range can be modified to show information from the last 1-24 hours or a custom date and time range.
The performance service is enabled at the cluster level. The performance history database is stored as a Virtual SAN object independent of vCenter. A storage policy is assigned to the object to control space consumption, availability, and performance of that object. If the object becomes unavailable, performance history for the cluster cannot be viewed until access to the object is restored.
The performance service is turned off by default. A few mouse clicks are all that is needed to enable the service.
At the cluster level, the performance monitoring service shows performance metrics for virtual machines running on Virtual SAN. These metrics provide quick visibility to the entire Virtual SAN cluster. A number of graphs are included such as read and write IOPs, read and write throughput, read and write latency, and congestion.
Backend consumption stems from activities such as metadata updates, component builds, etc. For example, a virtual machine with a number of failures to tolerate set to 1 with a failure tolerance method of RAID-1 (Mirroring). For every write IO to a virtual disk, two are produced on the backend – a write to each component replica residing on two different hosts.
In addition to virtual machine consumption and backend performance metrics, disk group and individual disk performance information is available at the host level. Seeing metrics for individual disks eases the process of troubleshooting issues such as failed storage devices.
Virtual SAN performance information for individual virtual machines and virtual disks. Metrics include IOPS, throughput, and latency. The figure below shows virtual disk-level Virtual SCSI throughput and latencies for reads and writes.
Figure x. Virtual Disk Performance
Traditional storage models utilize LUNs or volumes. A LUN or a volume is commonly configured with a specific disk configuration such as RAID to provide a specific level of performance and availability. The challenge with this model is each LUN or volume is confined to providing only one level of service regardless of the workloads that it contains. This leads to the provisioning of numerous LUNs or volumes in an attempt to provide the right levels or storage services to each workload. Maintaining a large number of LUNs or volumes leads to management complexity. Deployment and management of workloads and storage in traditional storage environments can be time consuming and error prone.
Storage Policy Based Management (SPBM) enables precise control of the storage services. Similar to other storage solutions, Virtual SAN provides services such as availability level, striping for performance, and the ability to limit IOPS. Policies that contain one or more rules are created using the vSphere Web Client.
These policies are assigned to virtual machines and individual objects such as a virtual disk. Storage policies can easily be changed and/or reassigned if application requirements change. These changes are performed with no downtime and without the need to migrate (Storage vMotion) virtual machines from one LUN or volume to another. This approach makes it possible to assign and modify service levels based on specific application needs even though the virtual machines reside on the same datastore.
Virtual SAN features an extensive management API and multiple software development kits (SDKs) to provide IT organizations options for rapid provisioning and automation. Administrators and developers can orchestrate all aspects of installation, configuration, lifecycle management, monitoring, and troubleshooting of Virtual SAN environments. This is especially useful in large environments and geographically disbursed organizations to speed up deployment times, reduce operational costs, maintain standards, and orchestrate common workflows.
SDKs are available for several programming languages including .NET, Perl, and Python. They are available for download from VMware Developer Center and include libraries, documentation, and code samples. For example, this Python script can be used to generate Virtual SAN capacity information: Virtual SAN Capacity – Total and Free from vCenter
Virtual SAN APIs can also be accessed through vSphere PowerCLI cmdlets. IT administrators can automate common tasks such as assigning storage policies and checking storage policy compliance. Consider a repeatable task such as deploying or upgrading two-node Virtual SAN clusters at 100 retail store locations. Performing each one manually would take a considerable amount of time. There is also a higher risk of error leading to non-standard configurations and possibly downtime. vSphere PowerCLI can instead be used to ensure all of the Virtual SAN clusters are deployed with the same configuration. Lifecycle management, such as applying patches and upgrades, is also much easier when these tasks are automated.
This video demonstrates a number of operations from creating a new cluster to configuring Virtual SAN using just a few lines of vSphere PowerCLI code:
vSphere PowerCLI: Creating a Cluster and Configuring Virtual SAN
Virtual SAN (VSAN) is an object datastore with a mostly flat hierarchy of objects and containers (folders). Items that make up a virtual machine (VM) are represented by objects. These are the most common object types you will find on a VSAN datastore:
VM Home, which contains virtual machine configuration files and logs such as the VMX and NVRAM files
Virtual Disk (VMDK)
Delta Disk (snapshot)
Memory Delta, which is present when the checkbox to snapshot a VM’s memory is checked
There are a few other objects that might be found on a VSAN datastore such as the VSAN performance service database and VMDKs that belong to iSCSI targets.
Each object consists of one or more components. The number of components that make up an object depends primarily on a couple things: The size of the objects and the storage policy assigned to the object. The maximum size of a component is 255GB. If an object is larger than 255GB, it is split up into multiple components. The image below shows a 600GB virtual disk split up into three components.
In most cases, a VM will have a storage policy assigned that contains availability rules such as Number of Failures to Tolerate and Failure Tolerance Method. These rules will also affect the number of components that make up an object. As an example, let’s take that same 600GB virtual disk and apply the Virtual SAN Default Storage Policy, which uses the RAID-1 mirroring failure tolerance method and has the number of failures to tolerate set to one. The 600GB object with three components will be mirrored on another host. This configuration provides two full copies of the data distributed across two hosts so that the loss of a disk or an entire host can be tolerated. Below is an image showing the six components (three on each host). A seventh component, the witness, is created by VSAN to “break the tie” and achieve quorum in the event of a network partition between the hosts. The witness object is place on a third host.
In this last example of component placement, we take the same 600GB virtual disk and apply a storage policy with RAID-5 erasure coding (Failures to Tolerate = 1). The object now consists of four components – three data components and a parity component – distributed across the four hosts in the cluster. If disk or host containing any one of these components is offline, the data is still accessible. If one of these components are permanently lost, Virtual SAN can rebuild the lost data or parity component from the other three surviving components.
Virtual SAN requires a minimum number of hosts depending on the failure tolerance method and number of failures to tolerate (FTT) configurations. For example, a minimum of three hosts are needed for FTT=1 with RAID-1 mirroring. A minimum of four hosts are required for FTT=1 with RAID-5 erasure coding. More implementation details and recommendations can be found in the Virtual SAN Design and Sizing Guide.
“Fault domain” is a term that comes up fairly often in availability discussions. In IT, a fault domain usually refers to a group of servers, storage, and/or networking components that would be impacted collectively by an outage. A very common example of this is a server rack. If a top-of-rack (TOR) switch or the power distribution unit (PDU) for a server rack would fail, it would take all of the servers in that rack offline even though the servers themselves are functioning properly. That server rack is considered a fault domain.
While the failure of a disk or entire host can be tolerated, what if all of these servers are in the same rack and the TOR switch goes offline? Answer: All hosts are isolated from each other and none of the objects are accessible. To mitigate this risk, the servers in a Virtual SAN cluster should be spread across server racks and fault domains must be configured in the Virtual SAN user interface. After fault domains are configured, Virtual SAN will redistribute the components across server racks to eliminate the risk of a rack failure taking multiple objects offline. This feature is commonly referred to as “Rack Awareness”. The diagram below shows what this might look like with a 12-node Virtual SAN cluster spread across four server racks.
Configuring Virtual SAN fault domains is quite simple as demonstrated in this video (no audio): Virtual SAN Fault Domains
Virtual SAN Stretched Clusters provide organizations with the capability to deploy a Virtual SAN cluster across two locations. These locations can be opposite sides of the same data center, two buildings on the same campus, or geographically disbursed between two cities. It is important to note that this technology does have bandwidth and latency requirements as detailed in the Virtual SAN Stretched Cluster Guide and 2-Node Guide.
A stretched cluster provides resiliency against larger scale outages and disasters by keeping two copies of the data – one at each location. If a failure occurs at either location, all data is available at the other location. vSphere HA restarts any virtual machines affected by the outage using the copy of the data at the surviving location. In the case of disaster avoidance such as an impending storm or rising flood waters, virtual machines can be migrated from one location to the other with no downtime using vMotion.
Since Virtual SAN is a clustering technology, a witness is required to achieve quorum in the case of a “split-brain” scenario where the two locations lose network connectivity. A Virtual SAN witness is simply a virtual machine running ESXi. The witness is deployed at a third location (separate from the two primary data locations) to avoid being affected by any issues that could occur at either of the main sites.
The witness does not store data such as virtual disk (VMDK) objects. Only witness objects are stored in the witness virtual appliance. If any one of the three sites goes offline, there is still more than 50% of each object’s components online to achieve quorum and maintain availability.
Enabling deduplication and compression can reduce the amount of physical storage consumed by as much as 7x, resulting in a lower total cost of ownership (TCO). Environments with highly-redundant data such as full-clone virtual desktops and homogenous server operating systems will naturally benefit the most from deduplication. Likewise, compression will offer more favorable results with data that compresses well such as text, bitmap, and program files. Data that is already compressed such as certain graphics formats and video files, as well as files that are encrypted, will yield little or no reduction in storage consumption from compression. In other words, deduplication and compression results will vary based on the types of data stored in an all-flash Virtual SANenvironment.
Deduplication and compression is a single cluster-wide setting that is disabled by default and can be enabled using a simple drop-down menu. Note that a rolling format of all disks in the Virtual SAN cluster is required, which can take a considerable amount of time. However, this process does not incur virtual machine downtime and can be done online, usually during an upgrade. Deduplication and compression are enabled as a unit. It is not possible to enable deduplication or compression individually.
Deduplication and compression are implemented after write acknowledgement to minimize impact to performance. Deduplication occurs when data is de-staged from the cache tier to the capacity tier of an all-flash Virtual SAN datastore. The deduplication algorithm utilizes a 4K-fixed block size and is performed within each disk group. In other words, redundant copies of a block within the same disk group are reduced to one copy, but redundant blocks across multiple disk groups are not deduplicated.
The compression algorithm is applied after deduplication has occurred just before the data is written tothe capacity tier. Considering the additional compute resource and allocation map overhead ofcompression, Virtual SAN will only store compressed data if a unique 4K block can be reduced to 2K orless. Otherwise, the block is written uncompressed to avoid the use of additional resources whencompressing and decompressing these blocks which would provide little benefit.
The processes of deduplication and compression on any storage platform incur overhead and potentially impact performance in terms of latency and maximum IOPS. Virtual SAN is no exception. However, considering deduplication and compression are only supported in all-flash Virtual SAN configurations, these effects are predictable in the majority of use cases. The extreme performance and low latency of flash devices easily outweigh the additional resource requirements of deduplication and compression. The space efficiency generated by deduplication and compression lowers the cost-per-usable-GB of all-flash configurations.
RAID-5/6 erasure coding is a space efficiency feature optimized for all-flash configurations. Erasure coding provides the same levels of redundancy as mirroring, but with a reduced capacity requirement. In general, erasure coding is a method of taking data, breaking it into multiple pieces and spreading it across multiple devices, while adding parity data so it may be recreated in the event one of the pieces is corrupted or lost.
Unlike deduplication and compression, which offer variable levels of space efficiency, erasure coding guarantees capacity reduction over a mirroring data protection method at the same failure tolerance level. As an example, let’s consider a 100GB virtual disk. Surviving one disk or host failure requires 2 copies of data at 2x the capacity, i.e., 200GB. If RAID-5 erasure coding is used to protect the object, the 100GB virtual disk will consume 133GB of raw capacity – a 33% reduction in consumed capacity versus RAID-1 mirroring.
While erasure coding provides significant capacity savings over mirroring, understand that erasure coding requires additional processing overhead. This is common among any storage platform today. Erasure coding is only supported in all-flash Virtual SAN configurations. Therefore, performance impact is negligible in most use cases due to the inherent performance of flash devices. Also note that RAID-5 erasure coding (FTT=1) requires a minimum of four hosts and RAID-6 erasure coding requires a minimum of six hosts.
Virtual SAN has the ability to limit the number of IOPS a virtual machine or virtual disk generates. There are situations where it is advantageous to limit the IOPS of one or more virtual machines. The term “noisy neighbor” is often used to describe when a workload monopolizes available I/O or other resources, which negatively impact other workloads or tenants in the same environment.
An example of a possible noisy neighbor scenario is month-end reporting. Management requests delivery of these reports on the second day of each month so the reports are generated on the first day of each month. The virtual machines that run the reporting application and database are dormant most of the time. Running the reports take just a few hours, but this generates very high levels of storage I/O. The performance of other workloads in the environment are impacted while the reports are running. To remedy this issue, an administrator creates a storage policy with an IOPS limit rule and assigns the policy to the virtual machines running the reporting application and database. The IOPS limit eliminates the performance impact to the other virtual machines. The reports take longer, but they are still finished in plenty of time for delivery the next day.
Keep in mind storage policies can be dynamically created, modified, and assigned to virtual machines. If an IOPS limit is proving to be too restrictive, simply modify the existing policy or create a new policy with a different IOPS limit and assign it to the virtual machines. The new or updated policy will take effect just moments after the change is made.
Virtual SAN is optimized for modern all-flash storage with space efficiency features such as deduplication, compression, and erasure coding that lower TCO while delivering incredible performance. Hardware expenditures are further reduced for 2-node cluster configurations using directly connected crossover cables. This lowers network switch costs, reduces complexity, and improves reliability especially for use cases such as remote offices. The Virtual SAN iSCSI target service enables physical servers and application cluster workloads to utilize a Virtual SAN datastore. The performance and health services make it easy to verify Virtual SAN configurations and closely monitor key metrics including IOPs, throughput, and latency at the cluster, host, virtual machine, and virtual disk levels. Quality of service can be managed by using IOPs limits on a per-virtual machines and per-virtual disk basis. All of these services are precisely managed using VM-centric storage policies. Virtual SAN scales to 64 nodes per cluster with up to 150k IOPS per node using the latest hardware innovations such as NVMe. Deployment options include Virtual SAN Ready Nodes, Dell EMC VxRail turnkey appliances, and build-your-own. All of these utilize components certified by VMware and the leading hardware OEMs to simplify hyper-converged infrastructure deployment. This provides organizations a vast number of options to run any app, any scale on an enterprise-class platform powered by Virtual SAN.