Posts in Category: vSAN 6

VMware vSAN: Politique de Stockage SPBM (Striping)

Number of disk stripes per object (Striping)

Cette variable permet de diviser les objets en plusieurs parties et de les répartir sur plusieurs disques au sein du Cluster.

  • Une valeur supérieur à 1 peut améliorer les performances, mais une augmentation des ressources consommées.
  • Valeur par défaut: 1.
  • Valeur maximale: 12.

Example : StripeWidth=3, NumberOfFailuresToTolerate (FTT) = 1

Prenant en compte un Cluster de 3 nœuds et chaque nœud possède un SSD pour le “Cache Tiers” et 3 disques mécaniques pour le “Capacity Tiers”. Cette configuration est effectivement juste car pour la première copie de la VM, les 3 “Stripes” seront partagés sur trois disques appartenant au même nœud et pour le Replica de la VM, les 3 “Stripes” seront répartis sur les 3 disques d’un second nœuds du Cluster.

Si pour la même configuration vous configurez le StripeWidth=4, les “Stripes” d’un même objet pourrons se retrouver sur plusieurs nœuds et donc la variable NumberOfFailuresToTolerate (FTT) = 1, ne pourra plus être respectée car les datas d’une même VM pourront être à cheval sur plusieurs nœuds..

VMware vSAN Stretched Cluster & 2 Node Guide

Introduction

VMware Virtual  SAN  6.1,  shipping  with  vSphere  6.0 Update 1, introduced a  new  feature called VMware  Virtual SAN Stretched  Cluster.  Virtual  SAN  Stretched  Cluster is a specific configuration implemented in environments where disaster/downtime  avoidance  is a key requirement.  This  guide was developed to provide additional insight and information for installation,  configuration  and  operation  of a Virtual  SAN  Stretched Cluster infrastructure  in conjunction  with VMware vSphere. This guide will explain how vSphere handles specific failure scenarios  and  discuss  various  esign  considerations  and  operational  procedures.

Virtual  SAN  Stretched  Clusters with Witness Host refers to a deployment where a user sets up a Virtual SAN cluster with 2 active/active  sites with  an identical number  of ESXi hosts distributed evenly between the two sites. The sites are connected via a high bandwidth/low  latency link.

The third site hosting the Virtual SAN Witness Host is connected to both of the active/active  data-sites.  This connectivity can be via low andwidth/high latency links.

Each site is configured as a Virtual SAN Fault Domain. The nomenclature  used to describe a Virtual  SAN  Stretched  Cluster  configuration  is  X+Y+Z,  where  X is the number  of ESXi hosts at data  site A, Y is the number of ESXi hosts at data site B, and Z is the number of witness hosts at site C. Data  sites are where virtual  machines  are  deployed.  The minimum supported configuration  is 1+1+1(3 nodes). The maximum  configuration  is 15+15+1
(31 nodes). In Virtual  SAN  Stretched  Clusters,  there is only one witness host in any configuration.

A virtual machine deployed on a Virtual  SAN  Stretched  Cluster will have one copy of its data on site A, a second copy of its data on site B    and any witness components  placed  on  the  witness  host in site C. This configuration is achieved through  fault domains alongside hosts  and  VM  groups,  and affinity rules. In the event of a complete site failure, there will be a full copy of the virtual  machine data as well as greater  than 50% of the components available. This will allow the virtual machine  to remain available on the Virtual SAN datastore.  If the virtual machine needs to be restarted on the other site, vSphere HA will handle this task.

vSphere Versions

Virtual  SAN  Stretched  Cluster  configurations  require  vSphere  6.0  Update  1 (U1) or greater.  This implies both vCenter Server 6.0 U 1 and ESXi  6.0 U1. This version  of  vSphere  includes  Virtual  SAN  version  6.1.  This  is  the  minimum version required for Virtual SAN Stretched Cluster  support.

vSphere & Virtual SAN

Virtual  SAN  version  6.1  introduced  features  including  both  All-Flash  and  Stretched  Cluster  functionality.  There  are  no  limitations  on  the  edition  of  vSphere  used  for  Virtual  SAN.  However,  for  Virtual SAN  Stretched  Cluster  functionality, vSphe re DRS is very desi rable . DRS will provide initial placement assistance, and will also  automatically migrate virtual machines to their co rrect site in accordance to Host/VM affinity rules. It can also help will locating virtual machines  to  their  co rrect  site  when  a  site  recovers  after  a  failure.  Otherwise  the administrator will have to manually carry out these tasks. Note that DRS is only available  in Enterprise  edition and  higher of vSphere.

Hybrid and All-Flash Support

Virtual SAN Stretched Cluster is supported on both hybrid configurations (hosts with local storage comprised of both magnetic  disks for capacity  and flash devices for cache) and  all-flash  configurations  (hosts with local storage made up of flash devices for capacity  and flash devices for cache).

On-disk Formats

VMware  supports  Virtual SAN Stretched Cluster with the v2 on-disk format only. The v1 on-disk  format  is based  on VMFS and  is the original on-disk  format  used for Virtual  SAN.  The v2 on-disk  format  is  the  version which comes  by default with Virtual SAN version 6.x. Customers  that upgraded from the original Virtual SAN 5.5 to Virtual SAN 6.0 may not have upgraded  the on-disk format for v1 to v2, and are thus still using v1. VMware ecommends upgrading the on-disk format to v2 for improved performance  and scalability,  as well as stretched  cluster  support.  In Virtual SAN 6.2 clusters,  the v3 on-disk  format  allows for additional features, discussed later, specific to 6.2.

Witness Host as an ESXi VM

Both physical ESXi hosts and virtual ESXi hosts (nested ESXi) are supportedfor the  witness host. VMware provides a Witness Appliance for those customers who wish to use the ESXi VM. A witness host/VM cannot  be shared between multiple Virtual  SAN  Stretched  Clusters.

Features Supported on VSAN but not VSAN Stretched

The following are a list of products and features support on Virtual SAN but not on a stretched cluster implementation of Virtual SAN.

    • SMP-FT, the new Fault Tolerant VM mechanism introduced in vSphere 6.0, is supported on standard VSAN 6.1 deployments, but it is not supported on stretched cluster VSAN deployments at this time. *The   exception to this rule, is when using 2 Node configurations in the same physical location.
    • The maximum value for NumberOfFailuresToTolerate in a Virtual SAN Stretched Cluster configuration is 1. This is the limit due to the maximum number of Fault Domains being 3.
    • In a Virtual SAN Stretched Cluster, there are only 3 Fault Domains. These are typically  referred to as the  Preferred, Secondary, and Witness Fault  Domains. Standard Virtual SAN configurations can be comprised of up to 32 Fault Domains.
    • The Erasure Coding feature introduced in Virtual SAN 6.2 requires 4 Fault Domains for  RAID5  type  protection and 6 Fault Domains for RAID6 type  protection. Because Stretched Cluster configurations only have 3 Fault Domains, Erasure Coding is not supported on Stretched Clusters at this time.

Features Supported on vMSC but not VSAN Stretched

The following  are a list of products and features support on vSphere Metro Storage Cluster (vMSC) but not on a stretched cluster implementation of Virtual SAN.

    • RR-FT, the original (and now deprecated) Fault Tolerant mechanism for virtual machines is supported on vSphere 5.5 for vMSC. It is not supported on Setched cluster Virtual SAN.
    • Note  that the new SMP-FT, introduced in vSphere 6.0 is not supported on either vMSC or stretched cluster VSAN, but does work on standard VSAN deployments.

Virtual SAN Stretched Clusters Versus Fault Domain

A common question is how stretched cluster differs from Fault Domains, which is a Virtual SAN feature that was introduced with Virtual SAN version 6.0.  Fault  domains enable what might be termed “rack awareness” where the components of virtual  machines could be distributed amongst multiple hosts in multiple racks, and should a rack failure event occur, the virtual machine would continue to be available. However, these racks would typically be hosted in the same data center, and if there was a data center wide event, fault domains would not be able to assist with virtual machines availability.

Stretched clusters essentially build on what fault domains did, and now provide what might be termed “data center awareness”. Virtual SAN  Stretched  Clusters can now provide availability for virtual machines even if a data center suffers a catastrophic outage.

The Witness Host

The witness host is a dedicated ESXi host (or appliance) whose purpose is to host  the  witness  component of virtual machines objects. The witness must have connection to both the master Virtual SAN node and the backup Virtual SAN node to join the cluster. In steady state operations, the master node resides in the “preferred site”; the backup node resides in the “secondary site”. Unless the witness host connects to both the master and the backup nodes, it will not join the Virtual SAN cluster.

Read Locality in Virtual SAN Stretched Cluster

In traditional Virtual SAN clusters, a virtual machine’s read operations are distributed across all replica copies of the data in the cluster.  In the case of a policy setting of NumberOfFailuresToTolerate=1, which results in two copies of the data, 50% of the reads will come from replica1 and 50% will come from replica2. In the case of a policy setting of Number Of Failures To Tolerate=2 in non-stretched  Virtual SAN clusters, results in three copies of the data, 33% of the reads will come from replica1, 33% of the reads will come from replica2 and 33% will come from replica3.

In a Virtual SAN Stretched Cluster, we wish to avoid increased latency caused by reading across the inter-site link. To  insure that 100% of reads, occur in the site the VM resides on, the read locality mechanism was introduced. Read locality overrides the NumberOfFailuresToTolerate=1 policy’s behavior to distribute reads across the two data sites.

DOM, the Distributed Object Manager in Virtual SAN, takes care of this. DOM is responsible for the creation of virtual machine storage objects in the Virtual SAN cluster. It is also responsible for providing distributed data access paths to these objects. There is a single DOM owner per object. There are 3 roles within DOM; Client, Owner and Component Manager. The DOM Owner coordinates access to the object, including reads, locking and object configuration and reconfiguration. All objects changes and writes also go through the owner. The DOM owner of an object will now take into account which fault domain the owner runs in a Virtual SAN Stretched Cluster configuration, and will read from the replica that is in the same domain.

There  is  now another consideration with this read locality. One must avoid unnecessary vMotion of the virtual machine between sites. Since the read cache blocks are stored on one site, if the VM moves around freely and ends up on the remote site, the cache will be cold on that site after the move. Now there will be sub-optimal  performance  until the cache is warm again. To avoid this situation,  soft affinity rules are used to keep the VM local to the same site/fault domain where possible. The steps to configure such rules will be shown  in detail in the vSphere DRS section of this guide.

Virtual SAN 6.2 introduced Client Cache, a mechanism that allocates 0.4% of host memory, up to 1GB, as an additional read cache tier. Virtual machines leverage the Client Cache of the host they are running on. Client Cache is not associated with Stretched Cluster read locality, and runs ndependently.

VMware vCenter Server

A Virtual SAN Stretched Cluster configuration can be created and managed by a single instance of VMware vCenter Server. Both the Windows version and the Virtual Appliance version (Linux) are supported for configuration and management of a Virtual SAN Stretched Cluster.

A Witness Host

Afficher l'image d'origine

In a Virtual SAN Stretched Cluster, the witness components are only ever placed on the witness host. Either a physical ESXi host or a special witness appliance provided by VMware, can be used as the witness host.

If a witness appliance is used for the witness host, it will not consume any of the customer’s vSphere licenses. A physical ESXi host that is used as a witness host will need to be licensed accordingly, as this can still be used to provision virtual machines should a customer choose to do so.

It is important that witness host is not added to the VSAN cluster. The witness host is selected during the creation of a Virtual SAN Stretched Cluster.

The witness appliance will have a unique identifier in the vSphere web client UI to assist with identifying that a host is in fact a witness appliance (ESXi in a VM). It is shown as a “blue” host, as highlighted below:

Note this is only visible when the appliance ESXi witness is deployed. If a physical host is used as the witness, then it does not change its appearance in the web client. A witness host is dedicated for each stretchd cluster.

Networking and Latency Requirements

When Virtual SAN is deployed in a stretched cluster across multiple sites using fault domains, there are certain networking requirements that must be adhered to.

Layer 2 and Layer 3 Support

https://i2.wp.com/blogs.vmware.com/virtualblocks/files/2016/01/Capture1.jpg?w=904

Both Layer 2 (same subnet) and Layer 3 (routed) configurations are  used in a recommended Virtual SAN Stretched Cluster deployment.

  • VMware recommends that Virtual SAN communication between the data sites be over stretched L2.
  • VMware recommends that Virtual SAN communication between the data sites and the witness site is routed over L3.

Note: A common question is whether L2 for Virtual SAN traffic across all sites is supported. There are some considerations with the use of a stretched
L2 domain between the data sites and the witness site, and these are discussed in further detail in the design considerations section of this guide. Another common question is whether L3 for VSAN traffic across all sites is supported. While this can work, it is not the VMware recommended  network  topology for Virtual SAN Stretched Clusters at this time.

Virtual SAN traffic between data sites is multicast. Witness traffic between a data site and the witness site is unicast.

Supported Geographical Distances

For VMware Virtual SAN Stretched Cluste rs, geographical distances are not a support concern. The key requirement is the actual latency numbers between sites.

Data Site to Data Site Network Latency

Data site to data site network refers to the communication between non-witness sites, in other words, sites that run virtual machines and hold virtual machine data. Latency or RTT (Round Trip Time) between sites hosting virtual machine objects should not be greater than 5msec (< 2.5msec one-way).

Data Site to Data Site Bandwidth

Bandwidth between sites hosting virtual machine objects will be workload dependent. For most workloads, VMware recommends a minimum of 10Gbps or greater bandwidth between sites. In use cases such as 2 Node configurations for Remote Office/Branch Office deployments, dedicated 1Gbps bandwidth can be sufficient with less than 10 Virtual Machines.

Please refer to the Design Considerations section of this guide for further details on how to determine bandwidth requirements.

Data Site to Witness Networklatency

This refers to the communication between non-witness  sites and the witness site.

In most Virtual SAN Stretched luster configurations, latency or RTT (Round Trip Time) between sites hosting VM objects and the witness nodes should not
be greater than 200msec (100msec one-way).

In typical 2 Node configurations, such as Remote Office/Branch Office deployments, this latency or RTT is supported up to 500msec (250msec one-way).

The latency to the witness is dependent on the number of objects in the cluster. VMware recommends that on Virtual SAN Stretched lusterconfigurations up to 10+10+1, a latency of less than or equal to 200 milliseconds is acceptable, although if possible, a latency of less than or equal to 100 milliseconds is preferred. For configurations that are greater than 10+10+1, VMware recommends a latency of less than or equal to 100 milliseconds is required.

Data Site to Witness Network Bandwidth

Bandwidth between sites hosting VM objects and the witness nodes are dependent on the number of objects residing on Virtual SAN. It is important to size data site to witness bandwidth appropriately for bo th availability and growth. A standard rule of thumb is 2Mbps for every 1000 objects on Virtual SAN.

Please refer to the Design Considerations section of this guide for further details on how to determine bandwidth requirements.

Inter-Site MTU Consistency

It is important to maintain a consistent MTU size between data nodes and the witness in a Stretched Cluster configuration. Ensuring that each VMkernel interface designated for Virtual SAN  traffic, is set to the same MTU size will prevent traffic fragmentation. The Virtual SAN Health Check checks for a uniform MTU size across the Virtual SAN data network, and reports on any inconsistencies.

Virtual Machines Per Host

The maximum  number of virtual machines per ESXi host is unaffected  by the Virtual SAN Stretched Cluster configuration. The maximum is the same as for normal VSAN deployments.

VMware recommends that customers should run their hosts at 50% of maximum  number of virtual machines supported in a standard Virtual SAN cluster to accommodate a full site failure. In the event of full site failures, the virtual machines on the failed site can be restarted  on the hosts in the surviving site.

Hosts Per Cluster

The minimum number of hosts in a Virtual SAN Stretched Cluster is 3. In such a configuration, site 1 will contain a single ESXi host,  site 2 will contain  a single ESXi host and then there is a witness host at the third site, the witness site. The nomenclature for such a configuration is 1+1+1. This is commonly  referred to as a 2 Node configuration.

The maximum number of hosts in a Virtual SAN Stretched Cluster is 31. Site 1 contains ESXi 15 hosts, site 2 contains 15 ESXi hosts, and the witness  host on the third site makes 31. This is referred to as a 15+15+1 configuration.

Witness Host

There is a maximum of 1 witness host per Virtual SAN Stretched Cluster. The witness host requirements are discussed in the design considerations section of this guide. VMware provides a fully supported witness virtual appliance, in Open Virtual Appliance (OVA) format, for customers who do not wish to dedicate a physical ESXi host as the witness. This OVA is essentially a pre-licensed ESXi host running in a virtual machine, and can be deployed on a physical ESXi host on the third site.

Number Of Failures to Tolerate

Because Virtual SAN Stretched Cluster configurations effectively have 3 fault domains, the Number Of Failures To Tolerate (FTT) policy setting, has a maximum of 1 for objects. Virtual SAN cannot comply with FTT values that are greater than 1 in a stretched cluster configuration.

Other policy settings are not impacted by deploying VSAN in a stretched cluster configuration and can be used as per a non-stretched  VSAN cluster.

Fault Domains

Fault domains play an important role in Virtual SAN Stretched Cluster. Similar to the Number Of Failures To Tolerate (FTT) policy setting discussed previously, the maximum number of fault domains in a Virtual SAN Stretched Cluster is 3. The first FD is the “preferred” data site, the second FD is the “secondary” data site and the third FD is the witness host site.

VMware vSAN Introduction and General Information

What Virtual SAN Delivers

Radically Simple Storage

Make your job easier by simplifying storage provisioning and management for vSphere. Deploy storage with just a few mouse clicks from the vSphere Web Client and enjoy native integration with the VMware stack. Virtual machine-centric storage policies automate storage services levels on a per-VM basis.

Advanced Availability and Management

Customers of all industries and sizes trust Virtual SAN to run their business-critical workloads, from key business applications to thousands of virtual desktops. Virtual SAN ensures that data is never lost if a disk, host, network or rack fails and can even tolerate entire site failures with synchronous replication and stretched clusters.

50% Lower TCO

Deploy on inexpensive industry-standard server components to remove large, upfront investments. Eliminate siloed, purpose-built hardware and automate management of storage service levels through VM-centric policies. Further improve TCO with storage efficiency features like deduplication and enhanced automation capabilities.

Exceptional Performance

Built on an optimized I/O data path in the hypervisor and designed for flash speeds, Virtual SAN delivers much better performance than a virtual appliance or external device. Experience up to 100,000 IOPs per host with all-flash and scale up to 64 hosts per cluster—a perfect match for virtual desktops, remote IT and business critical applications.

Enterprise-Ready Storage

Trust your mission-critical applications with the only hypervisor-embedded storage solution. Virtual SAN delivers all-flash performance with up to 100,000 IOPs per host, support for vSphere availability technologies like High Availability (HA), asynchronous replication, stretched cluster capabilities, and storage efficiency features for all-flash including deduplication, compression and erasure coding.

Linearly Scalable Storage

Virtual SAN delivers predictable, elastic and non-disruptive scaling of storage and compute resources, eliminating costly forklift upgrades. Every Virtual SAN cluster can scale out one node at a time or scale up by adding capacity to existing hosts and is capable of achieving over 8 PB of raw storage capacity.

FEATURES

All-Flash Performance

Virtual SAN is the only hyper-converged storage solution delivered directly from the hypervisor, resulting in a flash-optimized architecture that delivers as much as 100,000 IOPS per all-flash node.

Hypervisor-embedded architecture

Leverage the shortest, most efficient I/O data path for optimized performance with the least impact on CPU and memory resources.

All-Flash architecture

Deploy an all-flash architecture with flash-based caching and SSD data persistence to provide up to 100,000 IOPS per host with consistent, low latency.

Server-side caching

Improve virtual machine performance and minimize storage latencies by caching and accelerating read/write disk I/O traffic through enterprise-grade, server-side flash cache.

Self-tuning

Automatically rebuild and rebalance storage to align with the storage service levels assigned to each virtual machine in the cluster.

Quality of Service (QoS)

Automatically limits and monitors the IOPS consumed by specific virtual machines, eliminating noisy neighbor issues.

Storage Efficiency

Virtual SAN enables the most cost-efficient all-flash performance by delivering up to 10x greater storage utilization through data reduction technologies to optimize data footprint.

Deduplication and Compression

Optimizes all-flash storage capacity with as much as 7x data reduction while having minimal impact on server CPU and memory resources.

Erasure Coding

Increases usable storage capacity by up to 2x with the same high levels of data resiliency. Tolerate one or two failures with single parity or double parity protection.

Availability

Virtual SAN delivers enterprise availability for the most demanding business critical applications, capable of delivering 99.999% uptime and beyond with built-in and tunable failure tolerance settings.

Built-in failure tolerance

Maximize resiliency with built-in distributed RAID, cache mirroring, and per-VM control by specifying how many host, network, disk, or rack failures to tolerate.

Stretched Cluster

Enable enterprise-level availability that ensures no data loss and near zero downtime even in the event of an entire site failure with synchronous replication between two data centers.

Interoperability with the VMware stack

Leverage core vSphere features such as High Availability (HA) and Distributed Resource Scheduler (DRS) along with vSphere snapshots and VMware Site Recovery Manage.

5 Minute RPO with vSphere Replication

vSphere Replication for Virtual SAN provides asynchronous VM replication with RPOs of up to five minutes.

Management

Virtual SAN provides the simplicity of managing storage along with compute and networking in a single, tightly integrated interface–the vSphere Web Client.

Single-pane-of-glass management

Eliminate the need for training on specialized storage interfaces and tools by using vSphere Web Client-based management to provision storage in two clicks.

VM-centric policy-based control and automation

Deploy storage policies with a single click to automatically provision storage resources–no LUNs or RAID configurations necessary.

Health Service

Perform hardware, firmware, and driver compatibility checks directly from the vSphere Web Client, along with real-time diagnostics and reporting on performance and storage capacity.

GETTING STARTED

Virtual SAN is radically simple, enterprise-class software-defined storage powering VMware hyper-converged infrastructure. Get started by learning about Virtual SAN at your own pace or learn with us. We offer a variety of ways to experience the benefits of hyper-converged infrastructure (HCI) and Virtual SAN, from live webinars to self-paced on-demand courses.

Learn at Your Own Pace

Learn With Us

VMware vSAN Requirements and Compatability

What are the requirements for running Virtual SAN?

Virtual SAN has specific requirements for hardware (hosts, CPU, memory, number & type of storage devices), software (version of vCenter, vSphere configuration, ESXi host version & configuration), and networking.

In traditional configurations, Virtual SAN requires 3 hosts, with at least one disk group each, configured in a vSphere Cluster with Virtual SAN enabled, with a VMkernel port configured for Virtual SAN traffic that has connectivity to other hosts in the cluster. Note* Multicast traffic is required.

More detailed information for the different versions of Virtual SAN can be found in the relative KB article for each version of Virtual SAN.

Virtual SAN 6.0: http://kb.vmware.com/kb/2106708
Virtual SAN 5.5: http://kb.vmware.com/kb/2058424

What type of hardware is required for Virtual SAN?

Hosts that are certified to run VMware vSphere along with components that are certified to run Virtual SAN. Components certified for Virtual SAN are tested by the VMware Storage and Availability Business Unit.

Virtual SAN can be installed on hardware in a bring your own component, Ready Node, or Engineered Appliance offering.

Bring Your Own
The VMware vSphere Hardware Compatibility List details which hosts and CPU configurations are supported with ESXi.
The vSphere HCL can be found here: http://vmwa.re/hcl

When combined with the vSphere HCL, the Virtual SAN Compatibility Guide provides a list of hardware certified to run Virtual SAN.
The Virtual SAN Component HCL can be found here: http://vmwa.re/vsanhclc/

Ready Nodes
Ready Nodes are a pre-validated hardware options from various OEMs.
Ready Nodes can be found here: http://vmwa.re/vsanhcl/ 
The Ready Node selector can be found here: http://vsanreadynode.vmware.com/

Warning: Using uncertified hardware may lead to performance issues and/or data loss. The reason for this is that the behavior of uncertified hardware cannot be predicted. VMware cannot provide support for environments running on uncertified hardware.

What are the cluster requirements of Virtual SAN?

Cluster requirements include

  • A minimum of 3 nodes must contribute storage, unless using a 2 Node configuration along with an external witness.
    • 2 Node configurations were introduced in Virtual SAN 6.1.
  • All ESXi hosts must be managed by vCenter Server 6.0 and configured as a Virtual SAN cluster member.
  • vCenter Server must be at a release level equal to or higher than the ESXi hosts it is managing.
  • ESXi hosts in a VSAN cluster may not participate in any other cluster.

What are the memory requirements of Virtual SAN 6?

Memory requirements are determined by the number of disk groups and devices that are managed by ESXi.

Hosts should contain at least 32GB of RAM to accommodate the maximum number of disk groups and devices.

To calculate Virtual SAN 6.0 memory consumption , use this equation:

BaseConsumption + (NumDiskGroups x ( DiskGroupBaseConsumption + (SSDMemOverheadPerGB x SSDSize)))

Where:

  • BaseConsumption: This  is the fixed amount of memory consumed by Virtual SAN per ESXi host. This is currently 3 GB.
    • This memory is mostly used to house the VSAN directory, per host metadata, and memory caches.
  • NumDiskGroups: This is the number of disk groups in the host, should range from 1 to 5.
  • DiskGroupBaseConsumption: This is the fixed amount of memory consumed by each individual disk group in the host. This is currently 500 MB. This is mainly used to allocate resources used to support inflight operations on a per disk group level.
  • SSDMemOverheadPerGB: This is the fixed amount of memory we allocate for each GB of SSD capacity.
    • This is currently 2 MB in hybrid systems and is 7 MB for all flash systems.
    • Most of this memory is used for keeping track of blocks in the SSD used for write buffer and read cache.
  • SSDSize: The size of the SSD in GB.

KB Article 2113954 provides scenarios to better illustrate Virtual SAN memory requirements.

What are the processor requirements?

Processors that are approved on the vSphere Compatibility Guide are approved for use with Virtual SAN.

  • Virtual SAN typically consumes no more than 10% of CPU overhead per host for versions up to 6.1.
  • When using advanced Space Efficiency features in All-Flash architectures of Virtual SAN 6.2, may consume an additional 5% of CPU.

What are the software requirements?

The software requirements for Virtual SAN 6 are:

  • VMware vCenter Server must be at the same version or higher than the ESXi hosts it is managing.
  • ESXi hosts that participate in Virtual SAN Clusters must be version 6.0.
  • ESXi hosts participating in a VSAN 6.0 cluster must be running the same ESXi version.
  • Hosts versions may be mismatched during the duration of an upgrade.
  • When upgrading from Virtual SAN 5.5 to 6.0, the on-disk format must be upgraded to use all available features for the specific edition of Virtual SAN.
  • The VSAN-FS format for Virtual SAN 6.0 and 6.1 must be 2.0 to use all features of each edition.
  • The VSAN-FS format for Virtual SAN 6.2 must be 3.0 to use all features of 6.2.

What are the network requirements?

The networking requirements for Virtual SAN 6 are:

  • For hybrid configurations, each host must have a minimum of a single physical 1 GB Ethernet NIC available solely for Virtual SAN use.
  • For all flash configurations, each host must have a minimum of a single physical 10 GB Ethernet NIC available for Virtual SAN use. This NIC can be shared with other traffic.
  • Layer 2 multicast must be enabled on the physical switch connecting all hosts in the VSAN cluster.  Layer 3 multicast is supported.
  • Each ESXi host in the cluster must have a vmkernel port, regardless of whether it contributes to storage. For more information, see the Set Up a VMkernel Network for Virtual SAN section in the VMware Virtual SAN 6.0 Documentation.

Note: When running multiple Virtual SAN clusters, it is required to have isolated Virtual SAN networks.
This isolation can be achieved through the use of:

  • Each Virtual SAN network residing on a non-routed VLAN
  • Each Virtual SAN network with its own distinct Multicast addresses
    KB Article 2075451 covers setting Multicast addressing in Virtual SAN.

For more information, see Networking Requirements for Virtual SAN section in the VMware Virtual SAN 6.0 Documentation.

What are the storage requirements for Virtual SAN 6?

The storage requirements are:

  • For Caching:
    • At least 1 SAS/SATA Solid State Drive (SSD), PCIe flash disk, or other supported caching device.
    • Hybrid configurations will use 70% of the capacity of each caching device will be used for read caching.
    • Hybrid configurations will use 30%* of the capacity of each caching device will be used as a write buffer.
    • All-Flash configurations will use 100%* of the capacity of each caching device will be used as a write buffer.

*Up to 600GB

  • Capacity for virtual machine storage:
    • Hosts running in a hybrid cluster configuration must have at least 1 SAS, NL-SAS or SATA magnetic Hard Disk (HDD).
    • Hosts running in an all-flash disk group cluster configuration must have at least 1 SAS/SATA Solid State Drive (SSD), or a PCIe flash disk
  • Storage Controllers:
    • A SAS or SATA Host Bus Adapter (HBA), or RAID controller that is set up in non-RAID (pass through) or RAID 0 mode.
    • The HBA/RAID Controller must meet a minimum queue depth of 256. The Virtual SAN Compatibility Guide details recommendations for each certified controller.
    • Controller cache should be disabled or configured for 100% read.
  • Flash Boot Devices:
    • When booting a Virtual SAN 6.0 enabled ESXi host from a USB or SD card, the size of the disk must be at least 4 GB.
    • When booting a Virtual SAN host from a SATADOM device, you must use a Single-Level Cell (SLC) device and the size of the boot device must be at least 16 GB.

Notes:

  • Virtual SAN requires exclusive access to the local disks in the ESXi host.
  • Virtual SAN disks cannot be shared with another file system, such as Virtual Flash File System (VFFS), VMFS partitions, or an ESXi boot partition.
  • Do not format storage devices with VMFS or any other file system.
  • Ensure flash storage is not claimed by vSphere Flash Read Cache.
  • If you install ESXi on a USB or SD device and allocate all local storage to Virtual SAN, you do not have any local disk or datastore available for persistent logging. Configure a Dump Collector and a Syslog Collector to direct ESXi memory dumps and system logs to a server on the network, rather than to a local disk. For information, see the Configure ESXi Dump Collector with ESXCLIsection in the http://pubs.vmware.com/vsphere-60/topic/com.vmware.vsphere.install.doc/GUID-85D78165-E590-42CF-80AC-E78CBA307232.htmlvSphere Installation and Setup guide.

What are the licensing requirements for Virtual SAN 6?

Virtual SAN licensing varies by consumption type, version and edition

Virtual SAN may be used with any edition of vSphere. It is licensed by method and edition.

  • Licensing Method
    • For Virtual Desktop Infrastructure environments
      • Virtual SAN is licensed per desktop
      • Virtual SAN Advanced Edition licensing is included in VMware Horizon Advanced & Enterprise Suites.
    • For all other environments
      • Virtual SAN is licensed per CPU for each host consuming Virtual SAN resources
      • The capacity of the license must cover the total number of CPUs in the cluster.
    • For Remote Office Branch Office Deployments
      • Introduced in Virtual SAN 6.1, this edition allows for up to 25 virtual machines to be licensed at a remote or branch site.
      • ROBO licensing does not limit the number of hosts in the cluster, but rather the number of virtual machines to 25 per site.
      • A single Virtual SAN ROBO license can be spread across multiple sites, up to the 25 virtual machine limit.
      • More than total 25 virtual machines across multiple sites require additional ROBO licenses for each multiple of 25, with no more than 25 in a single site.
  • By Edition
    • Virtual SAN Standard licensing does not allow for All-Flash configurations
      • Storage Policy Based Management
      • Read/Write SSD Caching
      • Distributed RAID
      • Fault Domains
      • Snapshots/Clones
      • 6.1 added support for 5 minute RPO when replicating from one Virtual SAN datastore to another Virtual SAN datastore
      • 6.1 added support for health monitoring and vRealize Operations support with the Management Pack for Storage Devices
      • 6.1 added support for 2 Node configurations
      • 6.2 added support for software checksums
    • Virtual SAN Advanced licensing adds on Standard licensing features and supports for All-Flash configurations
      • 6.1 added for Stretched Cluster configurations
      • 6.2 removed support for Stretched Cluster configurations
      • 6.2 added support for cluster Deduplication & Compression for All-Flash configurations
      • 6.2 added support for Erasure Coding as a new Failure Tolerance Method for All-Flash configurations
    • Virtual SAN Enterprise licensing adds support for Quality of Service, and Stretched Cluster configurations
      • 6.2 added support for Stretched Cluster configurations
      • 6.2 added support for Quality of Service through IOPS limits

VMware vSAN 6.5 Technical Overview

Introduction to Virtual SAN

VMware Virtual SAN is radically simple, enterprise-class storage for hyper-converged infrastructure (HCI). Uniquely embedded in the VMware vSphere hypervisor, Virtual SAN delivers flash-optimized, high-performance storage. It leverages commodity x86 server components to drastically lower TCO by up to 50% and deliver all-flash solutions for as low as half the price of competitive hybrid HCI systems. Seamless integration with vSphere and the entire VMware stack makes it the simplest storage platform for business-critical workloads, virtual desktop infrastructure (VDI), remote office IT, disaster recovery, and DevOps infrastructures. Customers of all industries and sizes trust Virtual SAN to run their most important applications.

Key Benefits

  • Radically Simple – Configure with just a few clicks using the vSphere Web Client and automate management using storage policies. Eliminate expensive, complex, purpose-built storage array hardware and automate management of storage service levels through VM-centric policies.
  • High Performance – Deliver up to 150,000 IOPs per host and over 6 Million IOPs per cluster with predictable sub-millisecond response time from a single, all-flash configuration. Built on an optimized I/O data path designed for flash speeds, Virtual SAN delivers much better performance than virtual appliance and external device solutions.
  • Elastic Scalability – Easily grow storage performance and capacity by adding new nodes or drives without disruption. Linearly scale capacity and performance from 2 to 64 physical vSphere hosts per cluster.
  • Lower TCO – Lower storage costs by up to 50% by deploying standard x86 servers with local storage devices for low upfront investment and by shrinking data center footprint and operational overheads. Further improve total cost of ownership with features like deduplication, compression, erasure coding, iSCSI target services, and automation with PowerCLI.
  • Enterprise-Class Availability – Enable maximum levels of data protection and application availability with built-in tolerance for disk and host failures, stretched cluster configurations, and compatibility with DR solutions such as VMware Site Recovery Manager and vSphere Replication.
  • Advanced Management – Management for storage, compute and networking with advanced performance and capacity monitoring all within the vSphere Web Client.

Use cases for Virtual SAN include mission-critical applications, test and development, remote and branch offices, disaster recovery sites, and virtual desktop infrastructure (VDI). Virtual SAN supports a number of configuration options such as 2-node clusters for small implementations at remote offices to clusters as large as 64 nodes delivering over 6 million IOPS. Stretched clusters can also be configured to provide resiliency against site failure and workload migration with no downtime for disaster avoidance. VMware provides the market-leading HCI platform powered by vCenter Server, vSphere and Virtual SAN.

Separating Virtual SAN Witness Traffic

New in vSphere 6.5 and Virtual SAN 6.5 is the ability to separate witness network traffic from the network that connects the two physical hosts in a two-node cluster. This configuration option reduces complexity by eliminating the need to create and maintain static routes on the physical hosts. Security is also improved as the data network (between physical hosts) is completely separated from the WAN for witness traffic.

A Virtual SAN Witness is a virtual appliance that is utilized in stretched clusters and two-node configurations. The witness acts as a “tie-breaker” in certain events such as a network partition between hosts in a 2-node cluster. The witness virtual appliance stores metadata such as witness components. It does not store data components such as VM Home and virtual disk (VMDK) files.

Two-node VSAN cluster configurations are commonly used for branch offices and remote locations running a small number of virtual machines. A witness virtual appliance must not run on either of the two physical nodes in the cluster as this would defeat the purpose of the witness. The recommended location for the witness virtual appliance(s) is a primary data center connected to the remote locations by a WAN. Having the witness virtual appliances centrally located facilitates ease of management.

Two-Node Configuration with Crossover Cables

Physical Hosts in a two-node cluster can be connected using crossover cables. Organizations with existing 100Mbps or 1Gbps networking hardware can continue to utilize this equipment and enable 10Gbps connections for Virtual SAN and vMotion. All-flash Virtual SAN configurations (including two-node) require 10Gbps connectivity and vMotion performance is enhanced using faster connections between hosts. Connection reliability between the physical hosts is also improved with direct connections.

As with any storage fabric, redundant connections are recommended. In addition to providing resiliency against a NIC or cable failure, two connections between physical hosts enables administrators to separate Virtual SAN from vMotion traffic. It is possible for vMotion to saturate a 10Gbps link when migrating multiple virtual machine simultaneously, e.g., when a host is put into maintenance mode. This scenario has the potential to impact Virtual SAN performance while the migrations are taking place. The figure below shows one possible configuration that would prevent Virtual SAN and vMotion from using the same connection unless there is a NIC or cable failure.

For more information on network configuration options and recommendations, see the vSphere documentation and the Virtual SAN Network Design Guide.

iSCSI Target Service

Block storage can be provided to physical workloads using the iSCSI protocol. The Virtual SAN iSCSI target service provides flexibility and potentially avoids expenditure on purpose-built, external storage arrays. In addition to capital cost savings, the simplicity of Virtual SAN lowers operational costs.

The Virtual SAN iSCSI target service is enabled with just a few mouse clicks. CHAP and Mutual CHAP authentication is supported. Virtual SAN objects that serve as iSCSI targets are managed with storage policies just like virtual machine objects.

After the iSCSI target service is enabled, iSCSI targets and LUNs can be created. The screen shot below shows target and LUN configuration options.

The last step is adding initiator names or an initiator group, which controls access to the target, as shown here.

In nearly all cases, it is best to run workloads in virtual machines to take full advantage of Virtual SAN’s simplicity, performance, and reliability. However, for those use cases that truly need block storage, it is now possible to utilize Virtual SAN iSCSI Target Service.

Cloud Native Application Storage

New application architecture and development methods have emerged that are designed to run in today’s mobile-cloud era. For example, “DevOps” is a term that describes how these next-generation applications are developed and operated. “Container” technologies such as Docker and Kubernetes are a couple of the many solutions that have emerged as options for deploying and orchestrating these applications. VMware is embracing these new application types with a number products and solutions. Here are a few examples:

Photon OS – a minimal Linux container host optimized to run on VMware platforms.

Lightwave – an open source project comprised of enterprise-grade identity and access management services.

Photon Controller – a distributed, multi-tenant ESXi host controller optimized for containers.

Cloud native applications naturally require persistent storage just the same as traditional applications. Virtual SAN for Photon Controller enables the use of a Virtual SAN cluster in cloud native application environments managed by Photon Controller. Virtual SAN for Photon Controller includes developer-friendly APIs for storage provisioning and consumption. APIs and a graphical user interface (GUI) geared toward IT staff are also included for management and operations.

vSphere Integrated Containers Engine is a container runtime for vSphere, allowing developers familiar with Docker to develop in containers and deploy them alongside traditional virtual machine workloads on vSphere clusters. vSphere Integrated Containers Engine enables these workloads to be managed through the vSphere GUI in a way familiar to vSphere admins. Availability and performance features in vSphere and Virtual SAN can be utilized by vSphere Integrated Containers Engine just the same as traditional virtual machine environments.

Docker Volume Driver enables vSphere users to create and manage Docker container data volumes on vSphere storage technologies such as VMFS, NFS, and Virtual SAN. This driver makes it very simple to use containers with vSphere storage and provides the following key benefits:

  • DevOps-friendly API for provisioning and policy configuration.
  • Seamless movement of containers between vSphere hosts without moving data.
  • Single platform to manage – run virtual machines and containers side-by-side on the same vSphere infrastructure.

Virtual SAN along with the solutions above provides an ideal storage platform for developing, deploying, and managing cloud native applications.

New and Improved PowerCLI Cmdlets

VMware’s PowerCLI implementation is one of the most widely adopted extensions to the framework. It features a plethora of functions that abstract the vSphere API down to simple and powerful cmdlets including a number of cmdlets for Virtual SAN. This makes it easy to automate a number of actions from simply enabling Virtual SAN to deployment and configuration of a Virtual SAN stretched cluster. Here are a few examples of what can be accomplished with Virtual SAN and PowerCLI:

Assigning a Storage Policy to Multiple VMs with PowerCLI

Sparse Virtual Swap Files

Automated Deployments

With each new release of vSphere PowerCLI and Virtual SAN APIs, the functionality becomes more robust. Cmdlets have been added and improved for these and many other operations:

  • Enabling deduplication and compression
  • Enabling the performance service
  • Configuring the health check service
  • Creating all-flash disk groups
  • Fault domain management
  • Stretched cluster configuration
  • Selecting the data migration option for maintenance mode
  • Retrieve capacity information

vSphere administrators and DevOps shops can utilize these new cmdlets to lower costs by enforcing standards, streamlining operations, and enabling automation for vSphere and Virtual SAN environments.

512e Storage Devices

Disk drives have been using a native 512-byte sector size. Due to increasing demands for larger capacities, the storage industry introduced new formats that use 4KB physical sectors. These are commonly referred to as “4K native” drives or simply “4Kn” drives. Some 4Kn devices include firmware that emulates 512 byte (logical) sectors while the underlying (physical) sectors are 4K. These devices are referred to as “512B emulation” or “512e” drives.

vSphere 6.5 and Virtual SAN 6.5 support the use of 512e drives. The latest information regarding support for these new drive types can be found in this VMware Knowledge Base Article: Support statement for 512e and 4K Native drives for VMware vSphere and VSAN (2091600)

Servers with Local Storage

Virtual SAN clusters consist of any number of physical server hosts from two to 64. Each host contains flash devices (all-flash configuration) or a combination of magnetic disks and flash devices (hybrid configuration) that contribute cache and capacity to the Virtual SAN distributed datastore. Each host has one to five disk groups. Each disk group contains one cache device and one to seven capacity devices.

For all-flash configurations, the flash device(s) in the cache tier are used for write caching only as read performance from the capacity flash devices is more than sufficient. Two grades of flash devices are commonly used in an all-flash Virtual SAN configuration: Lower capacity, higher endurance devices for the cache layer and more cost effective, higher capacity, lower endurance devices for the capacity layer. Writes are performed at the cache layer and then de-staged to the capacity layer, only as needed. This helps extend the usable life of the lower endurance flash devices in the capacity layer.

In a hybrid configuration, one flash device and one or more magnetic drives are configured as a disk group. A disk group can have up to seven magnetic drives for capacity. One or more disk groups are utilized in a vSphere host depending on the number of flash devices and magnetic drives contained in the host. Flash devices serve as read and write cache for the Virtual SAN datastore while magnetic drives make up the capacity of the datastore. By default, Virtual SAN will use 70% of the flash capacity as read cache and 30% as write cache.

Deployment Options

Virtual SAN Ready Nodes are typically the easiest, most flexible approach when considering deployment methods. Virtual SAN Ready Nodes are x86 servers, available from all of the leading server vendors that have been pre-configured, tested and certified for Virtual SAN. Each Ready Node is optimally configured for Virtual SAN with the required amount of CPU, memory, network, I/O controllers and storage devices.

Turn-key appliances such as Dell EMC VxRail provide a fully-integrated, preconfigured, and pre-tested VMware hyper-converged solution for a variety of applications and workloads. Simple deployment enables customers to be up and running in as little as 15 minutes.

Custom configurations using jointly validated components from a number of OEM vendors is also an option. The Virtual SAN Hardware Quick Reference Guide provides some sample server configurations as directional guidance and all components should be validated using the VMware Compatibility Guide for Virtual SAN.

Network Considerations

All-flash Virtual SAN configurations require 10Gb network connectivity. 1Gb connections are supported for hybrid configurations although 10Gb is recommended. Currently, Virtual SAN requires the use of multicast on the network. Multicast communication is used for host discovery and to optimize network bandwidth consumption for the metadata updates. This eliminates the computing resource and network bandwidth penalties that unicast imposes in order to send identical data to multiple recipients.

Support for using crossover cables in a 2-node cluster is new in Virtual SAN 6.5. Utilizing crossover cables eliminates the need to procure and manage a 10Gb network switch for these hosts, which lowers costs – especially in scenarios such as remote office deployments. This configuration also improves availability and provides complete separation of Virtual SAN witness traffic from data traffic.

For more information on Virtual SAN network requirements and configuration recommendations, see the Virtual SAN Network Design Guide.

Storage Virtual Appliance Disadvantages

Storage in a hyper-converged infrastructure (HCI) requires compute resources that have been traditionally offloaded to dedicated storage arrays. Most HCI solutions require the deployment of storage virtual appliances to some or all of the hosts in the cluster to provide storage services to each host. These virtual appliances typically require CPU and/or memory reservations to avoid resource contention, which can result in performance degradation. Running a virtual appliance on every host in the cluster reduces the overall amount of compute resources available to run regular virtual machine workloads. Consolidation ratios will likely be lower and total cost of ownership rises when these storage virtual appliances are present and competing for the same resources as regular virtual machine workloads.

Storage virtual appliances can also introduce additional latency, which negatively affects performance. This is due to the number of steps required to handle and replicate write operations as shown in the figure below.

Virtual SAN Embedded in the vSphere Hypervisor

Virtual SAN does not require the deployment of storage virtual appliances or the installation of a vSphere Installation Bundle (VIB) on every host in the cluster. Virtual SAN is embedded in the vSphere kernel and typically consumes less than 10% of the compute resources on each host. Virtual SAN does not compete with other virtual machines for resources and the IO path is shorter.

A shorter IO path and the absence of resource-intensive storage virtual appliances enables Virtual SAN to provide extreme performance with minimal overhead. Higher consolidation ratios translate into lower total costs of ownership.

Enabling Virtual SAN

Virtual SAN is built into vSphere. Virtual SAN is enabled with just a few mouse clicks. There is no requirement to install additional software and/or deploy virtual storage appliances to every host in the cluster. Simply click the Enable Virtual SAN checkbox to start the process. Deduplication and compression can also be enabled at that time.

The next step is claiming local storage devices in each host for the Virtual SAN cache and capacity tiers. One or more disk groups are created in each host. Each disk group contains one cache device (flash) and one or more capacity devices (flash or magnetic). Virtual SAN pools these local storage device together to create a pool of shared storage.

The process of enabling Virtual SAN takes only a better of minutes. This is a tribute to the simplicity of Virtual SAN – especially when you compare it to other enterprise-class hyper-converged and traditional storage systems, which typically take much longer to set up.

Health Service

Virtual SAN 6.2 features a comprehensive health service that actively tests and monitors a number of items such as hardware compatibility, network connectivity, cluster health, and capacity consumption. The health service is enabled by default and configured to check the health of the Virtual SAN environment every 60 minutes.

The Health service is quite thorough in the number of tests it performs. As an example, proper network configuration is essential to a healthy Virtual SAN cluster and there are 11 tests in the“Network” section of the Virtual SAN Health user interface.

If an issue is detected, a warning is visible in the Virtual SAN user interface. Clicking on the warning provides more details about the issue. For example, a controller driver that is not on the hardware compatibility list (HCL) will trigger a warning. In addition to providing details about the warning, Virtual SAN Health also has an“Ask VMware” button, which brings up the relevant VMware Knowledge Base article.

vSphere and Virtual SAN support a wide variety of hardware configurations. The list ofhardware components and corresponding drivers that are supported with Virtual SAN can be found in the VMware Compatibility Guide. It is very important to use only hardware, firmware, and drivers found in this guide to ensure stability and performance. The list of certified hardware, firmware, and drivers is contained in a hardware compatibility list (HCL) database. Virtual SAN makes it easy to update this information, for use by the Health Service tests. If the environment has Internet connectivity, updates can be obtained directly from VMware. Otherwise, HCL updates can be downloaded to enable offline updates.

If an issue does arise that requires the assistance of VMware Support, it is easy to upload support bundles to help expedite the troubleshooting process. Clicking the “Upload Support Bundles to Service Request…” button enables an administrator to enter an existing support request (SR) number and upload the necessary logs with just a few mouse clicks.

Figure x. Virtual SAN HCL Database and Support Assistant

Proactive Tests

Virtual SAN proactive tests enable administrators verify Virtual SAN configuration, stability, and performance to minimize risk and confirm that the datastore is ready for production use. Three proactive tests are available:

VM creation test
Multicast performance test
Storage performance test.

The VM creation test creates and deletes a small virtual machine on each host confirming basic functionality. The multicast test verifies multicast is working properly on each host and performance meets Virtual SAN requirements. The following figure shows the results of a VM creation test.

The storage performance test is used to check the stability of the Virtual SAN cluster under heavy I/O load. There are a number of workload profiles that can be selected for this test as shown below. Keep in mind the storage performance test can affect other workloads and tasks. This test is intended to run before production virtual machine workloads are provisioned on Virtual SAN.

Capacity Reporting

Capacity overviews are available in the Virtual SAN user interface making it easy for administrators to see used and free space at a glance. Deduplication and compression information is also displayed.

Information is also available showing how much capacity various object types are consuming. Note that percentages are of used capacity, not of total capacity.

This list provides more details on the object types in the Used Capacity Breakdown chart:

Virtual disks: Virtual disk consumption before deduplication and compression
VM home objects: VM home object consumption before deduplication and compression
Swap objects: Capacity utilized by virtual machine swap files
Performance management objects: When the Virtual SAN performance service is enabled, this is the amount of capacity used to store the performance data
File system overhead: Capacity required by the Virtual SAN file system metadata
Deduplication and compression overhead: deduplication and compression metadata, such as hash, translation, and allocation maps
Checksum overhead: Capacity used to store checksum information
Other: Virtual machine templates, unregistered virtual machines, ISO files, and so on that are consuming Virtual SAN capacity

Performance Service

A healthy Virtual SAN environment is one that is performing well. Virtual SAN includes a number of graphs and data points that provide performance information at the cluster, host, virtual machine, and virtual disk levels. Time Range can be modified to show information from the last 1-24 hours or a custom date and time range.

The performance service is enabled at the cluster level. The performance history database is stored as a Virtual SAN object independent of vCenter. A storage policy is assigned to the object to control space consumption, availability, and performance of that object. If the object becomes unavailable, performance history for the cluster cannot be viewed until access to the object is restored.

The performance service is turned off by default. A few mouse clicks are all that is needed to enable the service.

Cluster Metrics

At the cluster level, the performance monitoring service shows performance metrics for virtual machines running on Virtual SAN. These metrics provide quick visibility to the entire Virtual SAN cluster. A number of graphs are included such as read and write IOPs, read and write throughput, read and write latency, and congestion.

Backend consumption stems from activities such as metadata updates, component builds, etc. For example, a virtual machine with a number of failures to tolerate set to 1 with a failure tolerance method of RAID-1 (Mirroring). For every write IO to a virtual disk, two are produced on the backend – a write to each component replica residing on two different hosts.

Host Metrics

In addition to virtual machine consumption and backend performance metrics, disk group and individual disk performance information is available at the host level. Seeing metrics for individual disks eases the process of troubleshooting issues such as failed storage devices.

Virtual Machine Metrics

Virtual SAN performance information for individual virtual machines and virtual disks. Metrics include IOPS, throughput, and latency. The figure below shows virtual disk-level Virtual SCSI throughput and latencies for reads and writes.

Figure x. Virtual Disk Performance

Traditional Storage Management

Traditional storage models utilize LUNs or volumes. A LUN or a volume is commonly configured with a specific disk configuration such as RAID to provide a specific level of performance and availability. The challenge with this model is each LUN or volume is confined to providing only one level of service regardless of the workloads that it contains. This leads to the provisioning of numerous LUNs or volumes in an attempt to provide the right levels or storage services to each workload. Maintaining a large number of LUNs or volumes leads to management complexity. Deployment and management of workloads and storage in traditional storage environments can be time consuming and error prone.

Storage Policy Based Management

Storage Policy Based Management (SPBM) enables precise control of the storage services. Similar to other storage solutions, Virtual SAN provides services such as availability level, striping for performance, and the ability to limit IOPS. Policies that contain one or more rules are created using the vSphere Web Client.

These policies are assigned to virtual machines and individual objects such as a virtual disk. Storage policies can easily be changed and/or reassigned if application requirements change. These changes are performed with no downtime and without the need to migrate (Storage vMotion) virtual machines from one LUN or volume to another. This approach makes it possible to assign and modify service levels based on specific application needs even though the virtual machines reside on the same datastore.

Automation

Virtual SAN features an extensive management API and multiple software development kits (SDKs) to provide IT organizations options for rapid provisioning and automation. Administrators and developers can orchestrate all aspects of installation, configuration, lifecycle management, monitoring, and troubleshooting of Virtual SAN environments. This is especially useful in large environments and geographically disbursed organizations to speed up deployment times, reduce operational costs, maintain standards, and orchestrate common workflows.

SDKs are available for several programming languages including .NET, Perl, and Python. They are available for download from VMware Developer Center and include libraries, documentation, and code samples. For example, this Python script can be used to generate Virtual SAN capacity information: Virtual SAN Capacity – Total and Free from vCenter

Virtual SAN APIs can also be accessed through vSphere PowerCLI cmdlets. IT administrators can automate common tasks such as assigning storage policies and checking storage policy compliance. Consider a repeatable task such as deploying or upgrading two-node Virtual SAN clusters at 100 retail store locations. Performing each one manually would take a considerable amount of time. There is also a higher risk of error leading to non-standard configurations and possibly downtime. vSphere PowerCLI can instead be used to ensure all of the Virtual SAN clusters are deployed with the same configuration. Lifecycle management, such as applying patches and upgrades, is also much easier when these tasks are automated.

This video demonstrates a number of operations from creating a new cluster to configuring Virtual SAN using just a few lines of vSphere PowerCLI code:

vSphere PowerCLI: Creating a Cluster and Configuring Virtual SAN

Objects

Virtual SAN (VSAN) is an object datastore with a mostly flat hierarchy of objects and containers (folders). Items that make up a virtual machine (VM) are represented by objects. These are the most common object types you will find on a VSAN datastore:

VM Home, which contains virtual machine configuration files and logs such as the VMX and NVRAM files
VM Swap
Virtual Disk (VMDK)
Delta Disk (snapshot)
Memory Delta, which is present when the checkbox to snapshot a VM’s memory is checked

There are a few other objects that might be found on a VSAN datastore such as the VSAN performance service database and VMDKs that belong to iSCSI targets.

Components

Each object consists of one or more components. The number of components that make up an object depends primarily on a couple things: The size of the objects and the storage policy assigned to the object. The maximum size of a component is 255GB. If an object is larger than 255GB, it is split up into multiple components. The image below shows a 600GB virtual disk split up into three components.

In most cases, a VM will have a storage policy assigned that contains availability rules such as Number of Failures to Tolerate and Failure Tolerance Method. These rules will also affect the number of components that make up an object. As an example, let’s take that same 600GB virtual disk and apply the Virtual SAN Default Storage Policy, which uses the RAID-1 mirroring failure tolerance method and has the number of failures to tolerate set to one. The 600GB object with three components will be mirrored on another host. This configuration provides two full copies of the data distributed across two hosts so that the loss of a disk or an entire host can be tolerated. Below is an image showing the six components (three on each host). A seventh component, the witness, is created by VSAN to “break the tie” and achieve quorum in the event of a network partition between the hosts. The witness object is place on a third host.

In this last example of component placement, we take the same 600GB virtual disk and apply a storage policy with RAID-5 erasure coding (Failures to Tolerate = 1). The object now consists of four components – three data components and a parity component – distributed across the four hosts in the cluster. If disk or host containing any one of these components is offline, the data is still accessible. If one of these components are permanently lost, Virtual SAN can rebuild the lost data or parity component from the other three surviving components.

Virtual SAN requires a minimum number of hosts depending on the failure tolerance method and number of failures to tolerate (FTT) configurations. For example, a minimum of three hosts are needed for FTT=1 with RAID-1 mirroring. A minimum of four hosts are required for FTT=1 with RAID-5 erasure coding. More implementation details and recommendations can be found in the Virtual SAN Design and Sizing Guide.

Fault Domains

“Fault domain” is a term that comes up fairly often in availability discussions. In IT, a fault domain usually refers to a group of servers, storage, and/or networking components that would be impacted collectively by an outage. A very common example of this is a server rack. If a top-of-rack (TOR) switch or the power distribution unit (PDU) for a server rack would fail, it would take all of the servers in that rack offline even though the servers themselves are functioning properly. That server rack is considered a fault domain.

Rack Awareness

While the failure of a disk or entire host can be tolerated, what if all of these servers are in the same rack and the TOR switch goes offline? Answer: All hosts are isolated from each other and none of the objects are accessible. To mitigate this risk, the servers in a Virtual SAN cluster should be spread across server racks and fault domains must be configured in the Virtual SAN user interface. After fault domains are configured, Virtual SAN will redistribute the components across server racks to eliminate the risk of a rack failure taking multiple objects offline. This feature is commonly referred to as “Rack Awareness”. The diagram below shows what this might look like with a 12-node Virtual SAN cluster spread across four server racks.

Configuring Virtual SAN fault domains is quite simple as demonstrated in this video (no audio): Virtual SAN Fault Domains

Stretched Clusters

Virtual SAN Stretched Clusters provide organizations with the capability to deploy a Virtual SAN cluster across two locations. These locations can be opposite sides of the same data center, two buildings on the same campus, or geographically disbursed between two cities. It is important to note that this technology does have bandwidth and latency requirements as detailed in the Virtual SAN Stretched Cluster Guide and 2-Node Guide.

A stretched cluster provides resiliency against larger scale outages and disasters by keeping two copies of the data – one at each location. If a failure occurs at either location, all data is available at the other location. vSphere HA restarts any virtual machines affected by the outage using the copy of the data at the surviving location. In the case of disaster avoidance such as an impending storm or rising flood waters, virtual machines can be migrated from one location to the other with no downtime using vMotion.

Since Virtual SAN is a clustering technology, a witness is required to achieve quorum in the case of a “split-brain” scenario where the two locations lose network connectivity. A Virtual SAN witness is simply a virtual machine running ESXi. The witness is deployed at a third location (separate from the two primary data locations) to avoid being affected by any issues that could occur at either of the main sites.

The witness does not store data such as virtual disk (VMDK) objects. Only witness objects are stored in the witness virtual appliance. If any one of the three sites goes offline, there is still more than 50% of each object’s components online to achieve quorum and maintain availability.

Deduplication and Compression

Enabling deduplication and compression can reduce the amount of physical storage consumed by as much as 7x, resulting in a lower total cost of ownership (TCO). Environments with highly-redundant data such as full-clone virtual desktops and homogenous server operating systems will naturally benefit the most from deduplication. Likewise, compression will offer more favorable results with data that compresses well such as text, bitmap, and program files. Data that is already compressed such as certain graphics formats and video files, as well as files that are encrypted, will yield little or no reduction in storage consumption from compression. In other words, deduplication and compression results will vary based on the types of data stored in an all-flash Virtual SANenvironment.

Deduplication and compression is a single cluster-wide setting that is disabled by default and can be enabled using a simple drop-down menu. Note that a rolling format of all disks in the Virtual SAN cluster is required, which can take a considerable amount of time. However, this process does not incur virtual machine downtime and can be done online, usually during an upgrade. Deduplication and compression are enabled as a unit. It is not possible to enable deduplication or compression individually.

Deduplication and compression are implemented after write acknowledgement to minimize impact to performance. Deduplication occurs when data is de-staged from the cache tier to the capacity tier of an all-flash Virtual SAN datastore. The deduplication algorithm utilizes a 4K-fixed block size and is performed within each disk group. In other words, redundant copies of a block within the same disk group are reduced to one copy, but redundant blocks across multiple disk groups are not deduplicated.

The compression algorithm is applied after deduplication has occurred just before the data is written tothe capacity tier. Considering the additional compute resource and allocation map overhead ofcompression, Virtual SAN will only store compressed data if a unique 4K block can be reduced to 2K orless. Otherwise, the block is written uncompressed to avoid the use of additional resources whencompressing and decompressing these blocks which would provide little benefit.

The processes of deduplication and compression on any storage platform incur overhead and potentially impact performance in terms of latency and maximum IOPS. Virtual SAN is no exception. However, considering deduplication and compression are only supported in all-flash Virtual SAN configurations, these effects are predictable in the majority of use cases. The extreme performance and low latency of flash devices easily outweigh the additional resource requirements of deduplication and compression. The space efficiency generated by deduplication and compression lowers the cost-per-usable-GB of all-flash configurations.

RAID-5/6 Erasure Coding

RAID-5/6 erasure coding is a space efficiency feature optimized for all-flash configurations. Erasure coding provides the same levels of redundancy as mirroring, but with a reduced capacity requirement. In general, erasure coding is a method of taking data, breaking it into multiple pieces and spreading it across multiple devices, while adding parity data so it may be recreated in the event one of the pieces is corrupted or lost.

Unlike deduplication and compression, which offer variable levels of space efficiency, erasure coding guarantees capacity reduction over a mirroring data protection method at the same failure tolerance level. As an example, let’s consider a 100GB virtual disk. Surviving one disk or host failure requires 2 copies of data at 2x the capacity, i.e., 200GB. If RAID-5 erasure coding is used to protect the object, the 100GB virtual disk will consume 133GB of raw capacity – a 33% reduction in consumed capacity versus RAID-1 mirroring.

While erasure coding provides significant capacity savings over mirroring, understand that erasure coding requires additional processing overhead. This is common among any storage platform today. Erasure coding is only supported in all-flash Virtual SAN configurations. Therefore, performance impact is negligible in most use cases due to the inherent performance of flash devices. Also note that RAID-5 erasure coding (FTT=1) requires a minimum of four hosts and RAID-6 erasure coding requires a minimum of six hosts.

Limiting IOPS

Virtual SAN has the ability to limit the number of IOPS a virtual machine or virtual disk generates. There are situations where it is advantageous to limit the IOPS of one or more virtual machines. The term “noisy neighbor” is often used to describe when a workload monopolizes available I/O or other resources, which negatively impact other workloads or tenants in the same environment.

An example of a possible noisy neighbor scenario is month-end reporting. Management requests delivery of these reports on the second day of each month so the reports are generated on the first day of each month. The virtual machines that run the reporting application and database are dormant most of the time. Running the reports take just a few hours, but this generates very high levels of storage I/O. The performance of other workloads in the environment are impacted while the reports are running. To remedy this issue, an administrator creates a storage policy with an IOPS limit rule and assigns the policy to the virtual machines running the reporting application and database. The IOPS limit eliminates the performance impact to the other virtual machines. The reports take longer, but they are still finished in plenty of time for delivery the next day.

Keep in mind storage policies can be dynamically created, modified, and assigned to virtual machines. If an IOPS limit is proving to be too restrictive, simply modify the existing policy or create a new policy with a different IOPS limit and assign it to the virtual machines. The new or updated policy will take effect just moments after the change is made.

Summary

Virtual SAN is optimized for modern all-flash storage with space efficiency features such as deduplication, compression, and erasure coding that lower TCO while delivering incredible performance. Hardware expenditures are further reduced for 2-node cluster configurations using directly connected crossover cables. This lowers network switch costs, reduces complexity, and improves reliability especially for use cases such as remote offices. The Virtual SAN iSCSI target service enables physical servers and application cluster workloads to utilize a Virtual SAN datastore. The performance and health services make it easy to verify Virtual SAN configurations and closely monitor key metrics including IOPs, throughput, and latency at the cluster, host, virtual machine, and virtual disk levels. Quality of service can be managed by using IOPs limits on a per-virtual machines and per-virtual disk basis. All of these services are precisely managed using VM-centric storage policies. Virtual SAN scales to 64 nodes per cluster with up to 150k IOPS per node using the latest hardware innovations such as NVMe. Deployment options include Virtual SAN Ready Nodes, Dell EMC VxRail turnkey appliances, and build-your-own. All of these utilize components certified by VMware and the leading hardware OEMs to simplify hyper-converged infrastructure deployment. This provides organizations a vast number of options to run any app, any scale on an enterprise-class platform powered by Virtual SAN.