[{"body":"","link":"https://blogs.hareeshkg.bid/","section":"","tags":null,"title":""},{"body":"Terms of Use The views expressed on this site are purely mine, and are in no way a reflection of the views of my employer. All suggestions and recommendations to the reader are based on what I have learned over past few years as a TSE and a SRE for vSAN. Troubleshooting steps and methods suggested on this blog should not be treated as recommendations from VMware Technical support. Some of the content discussed in the articles may be outdated as software, hardware, and architectures evolve.\nIf I state something inaccurately on any of the blog posts and if missed to quote a reference, please let me know, and I will be sure to correct it.\nAs described in the Author section I currently hold a position as a Site Reliability Engineer for vSAN at VMware by Broadcom, also a vExperts member. So it should be no surprise that the blog posts make look pretty favorably (but truthfully) about the products and solutions by VMware by Broadcom.\nAdvertisements To minimize conflicts of interest, this blog currently does not accept paid sponsorship advertisements. If this changes, the author does not accept any responsibility for the claims of a product or a service by a sponsor.\nCopyrights All Content contained in this Blog is owned by Hareesh K G, and may not be duplicated, copied, modified or adapted, in any way without our written permission. This Blog may contain service marks or trademarks of other companies and products in the form of words, graphics, and logos. Copying, redistribution, use or publication by you of any such Content, is strictly prohibited. Your use of the blog does not grant you any ownership rights to our Content.\n","link":"https://blogs.hareeshkg.bid/disclaimer/","section":"","tags":null,"title":"Disclaimer"},{"body":"Blog virtuallySensei is a personal blog about datacenter virtualization (SDDC) maintained by Hareesh K G. This focuses on troubleshooting guide, feature walkthrough, tips and tricks on Hyper-Converged-Solutions, Site recovery manager and other suite of products used in SDDC environments.\nAuthor Hareesh K G is a Site Reliability Engineer with VMware VSAN Engineering, his current focus is with VMware vSAN ® on-premises, his overall expertise is with Storage Availability Business Unit Products (VMware vSAN ®, VMware Site Recovery Manager® and vSphere Data Protection®). Started his career with EMC support for Clariion and VNX block storage in 2012 and has been with VMware since 2015.\nAchievements Advanced VMware Cloud Foundation 9.0 Storage VMware Certified Master Specialist – HCI 2023 VMware Certified Specialist – vSAN 2023 VMware Certified Professional – Data Center Virtualization 2023 VMware Certified Implementation Expert – Data Center Virtualization 2022 VMware Certified Advanced Professional – Data Center Virtualization Design 2022 VMware Certified HCI Master Specialist 2021 VMware Certified HCI Master Specialist 2020 VMware Certified Implementation Expert – Data Center Virtualization 2020 VMware Certified Advanced Professional – Data Center Virtualization Design 2020 VMware Certified Advanced Professional 6 – Data Center Virtualization Deployment VMware Certified Professional – Data Center Virtualization 2019 VMware Certified Professional – Network Virtualization 2019 vSAN Specialist 2019 Dell-EMC VxRAIL Xpert vSAN Specialist 2017 VMware VCAP5-DCA Vmware VCP-5 EMC VNX Specialist EMC Clariion Specialist EMC NAS Specialist EMC Information Storage and Management V2 Verify all Badges at Credly\n","link":"https://blogs.hareeshkg.bid/about/","section":"","tags":null,"title":"About"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/","section":"categories","tags":null,"title":"Categories"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/data/","section":"categories","tags":null,"title":"Data"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/esa/","section":"categories","tags":null,"title":"ESA"},{"body":"","link":"https://blogs.hareeshkg.bid/tags/index/","section":"tags","tags":null,"title":"Index"},{"body":"","link":"https://blogs.hareeshkg.bid/post/","section":"post","tags":["index"],"title":"Posts"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/protection/","section":"categories","tags":null,"title":"Protection"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/snapservice/","section":"categories","tags":null,"title":"SnapService"},{"body":"","link":"https://blogs.hareeshkg.bid/tags/","section":"tags","tags":null,"title":"Tags"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan/","section":"categories","tags":null,"title":"Vsan"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-data-protection/","section":"categories","tags":null,"title":"VSAN Data Protection"},{"body":"In this blog I will be walking through the basics of vSAN Data Protection and its deployment.\nWhat is vSAN Data Protection? With introduction of vSAN ESA 8.0 U3 VMware by Broadcom also announced vSAN Data Protection. vSAN Data Protection is a feature which is deployed through an OVF appliance that gives you additional local protection to your virtual machines running on vSAN ESA clusters with the help of native snapshots which were already supported in vSAN ESA.\nWhen a disaster is hit against your VMs running on your production vSAN clusters, you would like to have the VMs restored to its functional state as soon as possible. While restoring the affected from backup is an option or recovering the VMs to a different site with help of Site recovery manager + vSphere replication, you might wonder can this be done even faster.\nvSAN Data Protection brings you this functionality of local protection within your vSAN ESA clusters, by allowing you to create Protection Groups (PGs) very similar to what you are used to in SRM where you add a group of VMs into it and configure vSAN Data protection.vSAN Data protection uses local snapshots (also can be configured as immutable) to allow users to perform following:\nRevert existing VMs to one of the point in time (PIT) Snapshots on a given PG.\nClone a given snapshot to an instant clone VM to first check if the VM is functional and in an useful state before deciding to spin of the entire VM off that given snapshot\nRestore a VM which is no longer available on your datastore which has been deleted.\nCreate immutable snapshots for VMs which can never be deleted.\nPerform Ransomware recovery against affected VM on isolated network with use of VLCR with vSAN Data protection.\nDeploying vSAN Data Protection To deploy vSAN Data protection appliance (a.k.a SnapService Appliance) you will need to download the OVF from your Broadcom support portal under “My Downloads”, where you will find “VMware vSAN – VMware vSphere (vSAN Included) 8.0” see below Screenshot and have it accessible from your machine where you want to deploy it to the cluster.\n[ ![](/images/whats-new-with-vsan-data-protection-snap-service-appliance-SCR-20240809-kfzd.png) ](/images/whats-new-with-vsan-data-protection-snap-service-appliance-SCR-20240809-kfzd.png) Once the appliance is downloaded you can deploy it on to the cluster as shown in the video below:\nWhen the appliance is deployed successfully, you will see plugin registration tasks on the vCenter server for SnapService. If you dont see plugins getting registered on the vCenter then your OVF properties during the deployment was incorrect.\nIf you dont understand what is entered incorrectly during the OVF deployment , please login into your appliance and check the health of the docker containers and relevant logs else contact VMware by Broadcom support for assistance with the issue.\nvSAN Data Protection feature runs as docker services in the photon appliance, you can first check if all dockers are up and running by SSH to the vm.If any of the services were unhealthy or restarting continously you can refer to logs\n`root@snap docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 93840b53eade harbor-repo.vmware.com/snapservice/snapshot-service:8.0.3 \u0026quot;/snapshot-service -…\u0026quot; 7 hours ago Up 7 hours (healthy) 0.0.0.0:8088-8089-\u0026gt;8088-8089/tcp, :::8088-8089-\u0026gt;8088-8089/tcp, 0.0.0.0:8098-8099-\u0026gt;8098-8099/tcp, :::8098-8099-\u0026gt;8098-8099/tcp app-snapservice-1\ndaf5f1d83be8 harbor-repo.vmware.com/dockerhub-proxy-cache/library/postgres:14.2 \u0026quot;docker-entrypoint.s…\u0026quot; 7 hours ago Up 7 hours (healthy) 5432/tcp, 0.0.0.0:5433-\u0026gt;5433/tcp, :::5433-\u0026gt;5433/tcp app-mock-db-1\n7306b0a54936 harbor-repo.vmware.com/dockerhub-proxy-cache/library/postgres:14.2 \u0026quot;docker-entrypoint.s…\u0026quot; 7 hours ago Up 7 hours (healthy) 0.0.0.0:5432-\u0026gt;5432/tcp, :::5432-\u0026gt;5432/tcp app-db-1`\nLogs to investigate 1/var/log/vmware/snapservice/snap-service.log 2/var/log/vmware/snapservice/snap-service-panic.log You may also collect support bundles from vSAN Data protection appliance using known command “vm-support” via SSH session , export it via SCP session.\nThanks for taking time to read this post, I will cover more information on vSAN Data Protection in my future blogs.\n","link":"https://blogs.hareeshkg.bid/post/whats-new-with-vsan-data-protection-snap-service-appliance/","section":"post","tags":[],"title":"What’s new with vSAN Data Protection (Snap Service Appliance)"},{"body":"Introduction I have already talked about HCI Mesh (dHCI) when it was first launched with vSAN 7.0 U1 here, I would encourage you to read thru my previous blog for better understanding of this feature and its capabilities.\nThis feature was originally announced with vSAN OSA for 7.0 U1 with standard vSAN clusters where a server cluster could share its vSAN Datastore with 5 Client vSAN cluster and a Client Cluster could mount upto 5 remote vSAN Datastore.Compute-Only cluster was not supported in this initial release.\nOver a period of time with newer releases of vSAN until vSAN 8 we have had several new supported features\nMounting remote datastores supports one-way and two-way relationships\nAddressed multiple needs through access to hardware performance, capacity, cluster services, resilience via storage policy based management thereby providing interoperability between all types of standard OSA cluster with different datastore services. An administrator will be able to define the types of data service they are interested in which allows them to pick desired type of storage when deploying VMs.\nIt provided Scalability upto 128 hosts connected to remote vSAN datastore (hosts from server and client cluster included)\nClient cluster could connect upto 5 remote datastores\nServer cluster could serve upto 10 Client clusters.\nNo vSAN license needed for HCI Mesh compute clusters, they can mount upto 5 remote vSAN Datastores\nAdditional guided workflow and health checks for HCI Mesh clusters\nEasily mount or unmount remote vSAN datastore with assisted workflow and guardrails.\nHCI Mesh (dHCI)-Limits for vSAN until 8.0 GA vCenter Server license should be Enterprise or Enterprise plus.\nClient cluster: Number of remote vSAN datastores that it can connect to (was 5 in vSAN 7 U1 and remains this way for U2 \u0026amp; U3)\nServer cluster: Number of client clusters that a server cluster can serve (was 5 in vSAN 7 U1 and remains this way for U2 \u0026amp; U3. Improved to 10 in vSAN 8)\nDatastore host count totals. Number of hosts connecting to a vSAN datastore when counting hosts participating in client clusters, and server cluster (was 64 hosts in vSAN 7 U1 and is now 128 hosts for U2 \u0026amp; U3)\nHCI Mesh is only supported in vSAN 8 when using the Original Storage Architecture (OSA). In vSAN 8, HCI Mesh is not supported when using the Express Storage Architecture (ESA).\nLatency requirement between Server and Client clusters remains 5ms, VMware recommends to have latency less than 1ms for optimal performance.\nSeveral performance enhancements were made to vSAN sub-systems to ensure proper balance of object ownership and CPU usage on server cluster.\nvSAN VMKernel adapter used for inter host communication within the server cluster (local datastore) must be same for Client cluster vSAN communication, we cannot have separate vmkernel adapters like what we have for stretched cluster witness node traffic seperation.\nClusters running vSAN over RDMA is currently unsupported for HCI Mesh\nvCLS VMs spawned by EAM service on vCenter server required to keep vSphere HA services functional and it is not supported to run these on remote vSAN Datastore for a given client cluster. See https://kb.vmware.com/s/article/80877 for more details.\nWhats new in vSAN 8 U1 for HCI Mesh (Disaggregated HCI – dHCI) The formally known HCI Mesh name is slowly phased out and the term dHCI (Disaggregated HCI) is emphasized by VMware starting with vSAN 8 U1 the core functional features and concepts remains same as HCI Mesh which is now called with term disaggregated HCI.\nStorage Disaggregation (dHCI) with the Express Storage Architecture In vSAN 8 U1, disaggregation of storage (dHCI)is now compatible with the Express Storage Architecture. Users will be able to mount remote vSAN datastores living in other vSAN clusters, and also use an ESA cluster as the external storage resource for a vSphere cluster. Support of disaggregation when using the ESA maintains the interoperability and ease of use our customers have grown to appreciate with disaggregation in vSAN’s Original Storage Architecture.\nThere are still some limitations on how dHCI is supported for ESA and OSA below Table should clarify this.\n**Architecture****Server Cluster****Client Cluster****Supported**ESAStandard vSAN ClusterStandard vSAN ClusterYESESAStretched vSAN ClusterStandard vSAN ClusterNOESAStandard vSAN ClusterStretched vSAN ClusterNOESAStretched vSAN ClusterStretched vSAN ClusterNOOSAStandard vSAN ClusterStandard vSAN ClusterYESOSAStandard vSAN ClusterStretched vSAN ClusterNOOSAStretched vSAN ClusterStandard vSAN ClusterYESOSAStretched vSAN ClusterStretched vSAN ClusterYESESA/OSAStretched vSAN ClusterCompute-Only ClusterYes** **Compute-Only clusters can only mount vSAN Datastores of the same architecture exclusively, which means we cannot mount a mix of OSA and ESA datastore to compute-only clusters. A client cluster that is a compute-only cluster that is stretched over two sites. This exclusivity rule still applies to non compute-only clusters as well meaning an OSA client/server cluster can be mounted to one or more OSA clusters and viceversa for ESA clusters.\nYou will also be presented with a choice to choose the right network topology when presenting a remote datastore as a dHCI solution to a client cluster which is also a stretched cluster, lets assume that you have a “high bandwidth low latency” connection between the hosts in preffered site on server cluster and hosts in preferred site on Client cluster which means that both preferred site of two different cluster are on the same site , similar to this a secondary site on client cluster has a “high bandwidth low latency” connection to hosts in secondary site on server cluster. Then you must ensure that network topology is chosen accurately as “Asymmetric Connectivity” at the time of remote datastore mount operation, see below screenshots. If you have high bandwidth low latency connections between all sites, then you will choose “Symmetric Connectivity“\nSelect network topology dHCI – HCI Mesh – Stretched cluster\nConfigure Site coupling/mapping correctly when choosing Asymmetric setup\nSimilar to above steps when presenting a remote datastore from the server cluster which is a stretched cluster to a standard vSAN cluster or a compute only cluster , you will be asked to pick the correct topology to make sure that all VMs created on the compute-only cluster or a standard vSAN cluster will get the least latency from remote vSAN datastore.\nChoose right network topology when consuming a stretched cluster as remote dHCI/ HCI-Mesh datastore on client cluster\nConfigure SIte coupling between hosts in Client Cluster (Compute-only/Standard vSAN Cluster) with correct site to ensure low latency connection\nCross vCenter Sever consumption of remote vSAN Datastore We now have support for mounting remote vSAN Datastore (dHCI) across one or more vCenter servers for vSAN clusters running Original Storage Architecture.All existing workflows and pre-checks have been updated to accommodate disaggregation connectivity across vCenter Servers. To support such connections, it shall be assumed that participating clusters are running vSAN 8 U1 or greater, with all participating vCenter Servers running the very latest version. Mounting remote ESA vSAN Datastore cross vcenter functionality is yet not supported in this release.\nScaling limitation additions when using disaggregation across vCenter Servers. A single client vCenter can consume up to 5 server vCenter Servers for the purpose of serving datastore sources. A single server vCenter can be consumed as datastore sources by up to 10 client vCenter Servers, this very similar to the limitations within the same vCenter server.\nAdding Remote vSAN Datastore from different Server Vcenter Servers\nEnter VC-IP/FQDN and Credtentials\nAccept Certificate to proceed\nPass Compatibility Checks\nPost Remote vCenter server Configuration\nAttempting to add an ESA datastore from a remote vCenter server will will Compatibility check while adding a remote datastore at the cluster level. If the Client Cluster and Server cluster are running different ESXi versions or On-Disk-Format version Compatibility checks will prevent mounting of Remote Datastore.\n","link":"https://blogs.hareeshkg.bid/post/state-of-disaggregated-hci-hci-mesh-vsan-8-u1/","section":"post","tags":[],"title":"State of Disaggregated HCI (HCI Mesh)-vSAN 8 U1"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-8-u1/","section":"categories","tags":null,"title":"VSAN 8 U1"},{"body":"Introduction In vSAN 8.0, VMware vSAN introduced support for “Large Write Buffer”(LWB) cache disk size of upto 1.6TB can be used with OSA architecture. Although larger disks were supported in the previous releases for use in caching tier for vSAN OSA disk groups, the write buffer usage was limited to 600GB even when cache disks are larger, using larger disks in the previous releases (7.0 U3 and lower) was only preferred to increase endurance life of Cache Disk SSD.\nLarge Write Buffer feature (LWB) is targeted to support modern applications which have started to use large block size for writes (8k thru 256k), where such applications tends to perform bursty writes to vSAN which can easily fill up existing default write buffer of 600GB in Cache disk, which leads to sudden drop in write throughput and increased latency due to Congestion mechanism which is built into the system. Having write buffer of size upto 1.6TB can help absorb such bursty IO pattern better while providing consistent performance to all VMs in the cluster.\nThis Large Write Buffer (LWB) feature support is not enabled by default on vSAN 8.0 and higher version OSA clusters by default, allows users to decide to enable this feature. A host level advanced config option is used to enable/disable LWB, additionally we need to recreate every diskgroup post enabling/disabling the configuration flag for LWB to function. Please refer to official VMware KB for more details.\nPros: Provides significant performance improvement for Large block bursty workload which might easily fill up default 600GB write buffer. This will have better and improved sustained high throughput at Cache disks when using LWB feature.\nLWB feature keeps a lot of data set in the cache tier when there steady state workload, thereby reducing write amplification for IOs when it receives read modified writes to blocks which are yet to be destaged to capacity tier section.\nReduces load on the Capacity tier section because LWB handles most of the overwrites in the cache tier and de-stage to Capacity disks is not triggered too often, throttled destage is only triggered when 30% of write buffer is filled up, unthrottled destage is triggered only when nearing max write buffer usage.\nBetter coalescing and full stripe writes to the capacity tier disks with LWB since more overwrites are handled in Cache disks, reduces write amplification while destaging to Capacity Tier.\nLarge Write Buffer also significantly improves read performance since most of the reads will be served directly from the cache disk without needing to read from capacity disks.\nSignificantly improves resynchronization performance and ETA (IOPS/Throughput) because of LWB can accept more resync-writes while accepting writes from guests.\nSignificantly improves reboot times ~50% faster as LWB allows most/all of the in-memory metadata to be dumped in to the Cache disk, which can be replayed quickly during reboots directly instead of sequentially reading it from Cache/Capacity disks.\nCons: Increased memory requirement – Every vSAN disk group with large cache tier will incur an additional memory cost of 5GB. With the max number of supported disk groups per host (which is 5), the max additional memory cost increase is 25GB per host. Deployments that are tightly provisioned in terms of memory might experience issues due to this increased memory usage from vSAN.\nUnpredictable Performance with heterogeneous Setup – While the full benefits of the large cache tier may not be realized until all disk groups in a cluster are using the feature, there is no performance or functional regression from having a mix of small and large cache drives. Having mixed/heterogeneous sized Cache disks within a cluster with this feature enabled might lead to unpredictable performance.\nRequires all diskgroups to be recreated post enabling the feature, you will need to perform this task when there is minimal/steady state workload running to reduce the impact to guest performance due to resync traffic caused by Diskgroup recreation (Recommended to use full Data evacuation when you have enough free space in the cluster or Ensure Accessibility method recreation when there is space constraint in the vSAN Datastore)\nHow to enable/disable Large Write Buffer (LWB) Command to check the current setting: `esxcfg-advcfg -g /LSOM/enableLargeWb` Command to enable the feature: ``` `esxcfg-advcfg -s 1 /LSOM/enableLargeWb` ``` Please follow official VMware [KB-89485](https://kb.vmware.com/s/article/89485?lang=en_US), where steps are documented clearly. Prerequisite Read and understand Pros and Cons explained above. Ensure that each host has enough additional memory (5GB per DG) before enabling/recreating diskgroups and also ensure that you have enough memory for HA restarts for VMs when there are host failures.\nvCenter Server should be version 8.0 or higher.\nAll hosts in the cluster should be 8.0 or higher, make sure that all hosts are on the same build version before enabling the feature. (i.e not recommended to enable the feature with hosts in mix-mode some running 8.0.x and others 8.1.x)\nI highly recommend you to read thru VMware Official Blog on Large Write Buffer for additional information.\nIf you wish to read other related topics head to Home-Page, thanks for taking time to read this article on “vSAN 8 Large Write buffer (LWB) for OSA Architecture”.\n","link":"https://blogs.hareeshkg.bid/post/large-write-buffer-for-vsan-8-osa-architecture/","section":"post","tags":[],"title":"Large Write Buffer for vSAN 8 OSA Architecture"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-8/","section":"categories","tags":null,"title":"VSAN 8"},{"body":"vSANTraces are generally used to troubleshoot and debug critical performance issues in vSAN(OSA and ESA) and also in some cases they are used to troubleshoot vSAN data-path related problems such as Stuck-IO, Congestion, slow/no de-stage etcetera. vSANTraces are not require to troubleshoot/debug all kinds of issues however is required for certain cases.\nIn any vSAN enabled host, you might have noticed that the default location/path where the traces are collected is “/vsantraces”. Depending on the requirement, users may however change the path for vSANTraces to external Storage or a local datastore and change the default settings of “Number Of Files To Rotate:” and “Maximum Trace File Size:” for vSANTraces using esxcli commands. Having said that VMware doesnt recommend/allow customers to set the path for vSANTraces to vSAN-Datastore directly because it will impact performance of the virtual machines running on vSANDatastore as vsanTraces are write heavy (every subsystem of vSAN data-path writes to traces for bunch IO) which helps engineering to debug issues.\nvsantraces are very chatty and time period coverage/retention period is very short worst case less than 10mins. When you delay support bundle collection after an issue in your vSAN environment there is high probability that vsantraces have already rolled over. In such cases you might have to reproduce the issue if possible if you understand how to manifest it or wait for the issue to occur again and collect traces/support bundles immediately after the issue, so that VMware support and engineering can debug certain performance issues, congestion issues..etc.\nThe problem is that you might not know when such issue can happen again and it will be too difficult to manually monitor such issues and collect data for support and engineering.VMware introduced this new feature called “Native vSAN Trace object” for vSAN 8.0 U1 which helps to alleviate the problems which I have explained above.\nWhat does “Native vSAN Trace object” feature offer? Creates a NameSpace Object in vSAN Datastore (is hidden folder “.vsan.trace” ) with storage policy of FTT-1/RAID-1 on both OSA and ESA architecture.\nAutomatically creates multiple sub-directories under the namespace folder for each host with their respective vSAN-Host-UUIDs\nUnder each host-UUID folder it creates additional sub-directories with date format (YYYY-MM-DD)\nPeriodically dumps vsantraces from the default path /vsantraces under respective date sub-directory.\nPersists/retains traces files under each dated-sub-directories for upto 6 days and deletes older files.\nThey can use upto 512GB or 1% of vSAN Datastore at most. Usage of “Native Trace Object” can be viewed under “Capacity” section for vSAN. The Native trace Object is also displayed under “Virtual Objects” view with type “other” just like vSAN Performance Object.\nThis feature is enabled only if the trace log location remains under the default path/setting ( /vsantraces)\nOffers a command to create a support-bundle by dumping traces from native vsan trace object, which can be run from each host respectively.\nIn a stretched cluster setup native trace object feature only collects traces from data nodes and witness traces are excluded.\nPre-Requisite VMware vCenter server hosting a given vSAN cluster should be on build version 8.0 U1 or higher.\nAll hosts in the cluster should be on build version 8.0 U1 or higher.\nDefault setting for vsantrace should not be changed from /vsantraces for this feature to work.\nNative Trace Object in UI Capacity UI for vSAN\nVirtual Object View for vSAN\nNative trace Object CLI Check if default vsantraces settings:\nEnsure default vsantrace path is pointing to /vsantraces\nIdentify native trace object namespace folder:\nidentify .vsan.trace folder or UUID which matches virtual Objects view\nCheck the storage Policy for the native trace object since its not shown in virtual Object View:\nCheck Storage Policy for Native Trace Object\nViewing folder structure for each host inside Native Trace Object\nHost-Subfolder structure inside native trace Object\nSub-Folder Structure inside each host UUID\nWhen collecting regular support bundle trace files from Native trace object is not collected.\nvm-support command with specific manifest needs to be run on each host to collect traces from Native trace object for each host in the cluster.\nThis feature is intentionally implemented this way to make sure support bundles are not very large and we do not want vsantraces from native trace object file is not required for every other support ticket.\nSyntax: vm-support -a Storage:VSANTracesExtended -w \u0026lt;localdatastore path or /scratch/\u0026gt;\nHow to collect a support bundle which only contains native trace files\nCurrent Limitation Deleting old files from vsan trace object folder will not recover space on vSAN Datastore as it doesnot support unmap. vSAN native trace object keeps saving latest trace files and deletes the old ones. It prefers to request new disk space while adding the new files until reaching the double size of the object. After that, the object will reuse the old space if only no old space is available. VMConAWS users will also see native trace object being created as part of future upcoming VMC releases, however VMC users will not have direct access to this object or vSAN traces as its part of SaaS and VMware will manage this. Conclusion Native vSAN Trace object feature definitely helps to RCA a lot of unknown performance issues/congestion issues which cannot be reproduced, it is highly recommended to ensure this is fully functional after you upgrade to vSAN 8.0 U1 or higher version. Default vsantraces will be still available when a support bundle is collected. When VMware support requests older vsantraces while troubleshooting certain issues, then you may follow the steps described in this blog to collect and upload additional trace files from Native vSAN Trace Object.\nIf you want to read other related topics head to Home-Page, thanks for taking time to read this article on “Native vSAN Trace Object”.\n","link":"https://blogs.hareeshkg.bid/post/native-vsan-trace-object/","section":"post","tags":[],"title":"Native vSAN Trace Object"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-esa/","section":"categories","tags":null,"title":"VSAN-ESA"},{"body":"Introduction vSAN-ESA has introduced this concept of Storage Pools where all the disks in a given storage-pool contributes towards total capacity available for consumption. This initial release of vSAN ESA supports compression and encryption however doesn’t support Deduplication. vSAN ESA also introduced this concept of a log structured file System a.k.a LFS which I had discussed about briefly in my previous blog post, see My-take-on-ESA.\nWhen creating objects on vSAN-ESA datastore, the initial allocated space for each object by default is 3X the size of the object +** 2% of the Object Size**.\n2% is used for the** performance leg** in the Object layout . If you would like to know more about performance leg and Capacity legs, I suggest you to read thru my previous blog on vSAN-ESA LFS.\nThe Capacity leg is allocated space is 3X the object size+ 2% inclusive, in addition every child under the Capacity leg is always created in multiple of 765 GB, depending on the size of the object many such child legs are created.Finally both performance leg and capacity legs are concatenated at the top.\nGeneric Layout for a vSAN ESA object using RAID-6\nStripe Width on vSAN ESA vSAN-ESA is immune to stripe-width definition, by default vSAN-ESA tries to provide a stripe-width of 2 and any definition higher than value of 2 will be ignored. The reason for this logic is there will be no performance benefits to the objects. Users are still allowed to define higher stripe width just because the SPBM UI is shared between vSAN-OSA and vSAN-ESA, a vCenter server could be managing one or more ESA and OSA clusters at the same time, hence the policy definition permits stripe width definition more than 2 .\nThick Provisioning on vSAN ESA vSAN-ESA allows thick provisioning very similar to OSA. There are some myths circulating around this topic where thick provisioned objects provide better performance than thin provisioned objects, these claims are false. It is always recommended to use thin provisioning at all times and never use thick provisioning unless.\nRAID-1 on vSAN ESA vSAN-ESA allows users to create RAID-1 objects on 3 or more hosts in the cluster, number of host failure tolerance can be defined upto 3. Provisioned space for an Object with one host failure tolerance will be (3X Object Size + 2% of Object Size) x 2 mirrors.\nExample if the object size is 100GB, the allocated space for this object will be (1003)+(2% of 100)=302GB. Though it seems like we have over provisioned a lot of space, this object should not exceed total consumption of 202GB (1002 + 2% of 100) unless there were snapshots created on this objects and if they grow more than provisioned space.\nRAID-5 with 2+1, 3+1 , 4+1 hosts – ESA vSAN-ESA also allows users to deploy VMs with a RAID-5 storage policy on just 3 nodes (2 data + 1 Parity),the layout remains the same even for 4 hosts (3+1) this gives significant space savings which is just 1.5x the size of the object. vSAN OSA could never allow users to create a RAID-5 object with just 3 nodes, a RAID-5 object on OSA needs minimum of 4 hosts and uses 1.33x size of the object. ESA also allows a different layout (4 Data + Parity)when you have 6 or more hosts in the cluster, a RAID-5 object with such layout only consumes 1.25x its size. Hence ESA provides better overall space savings with erasure coding.\nExample : 1.5TB object created with 4 hosts (3+1) layout with RAID-5 SPBM\nIn the above example the 1.5TB object can consume upto 2.25TB (1.5TB*1.5 RAID-5 overhead) when object consumes all the space without snapshots. If the user creates snapshots when the object is approaching its full capacity additional 765GB child legs are spawned to accomodate snapshot growth. You will also see that the third RAID-0 child which is for parity is allocated with the same size as the data legs, this is expected and its just that it wont consume actual space like the data legs.\nExample: 1.5TB object created with 6 hosts (4+1+1 spare) layout with RAID-5 SPBM\nIn the above example this object can consume upto 1.875TB (1.5TB *1.25 RAID-5 overhead) when space is fully consumed. If the user creates snapshots while the object is approaching its full capacity additional 765GB child legs are spawned to accomodate snapshot growth. The parity child P under the 765GB leg is allocated with the same size as data legs however it will be only used to write parity data.\nWhen there is a host failure or when a cluster is downsized from 4+1 to a 3+1 layout or 2+1 layout , objects are automatically reconfigured adaptively, you might want to read thru “Adaptive RAID-5” section on the official VMware blogs.\nRAID-6 with 6 or more Hosts – ESA RAID-6 behaviour on vSAN ESA is same as OSA , which means any object created on vSAN-ESA will consume 1.5x the size of the object. The performance leg will have a total of 3 mirrored component in order to tolerate 2 host failure. Each Capacity leg child of 765GB will be constituted of 6 components of size 191.25GB (765/4) as we will have 4 data components and two parity components. These parity components will have the same allocated space of 191.25GB however will only consume space for parity. If no snapshots were created on the object this could grow upto 2.25TB (1.5TB*1.5 RAID-6 overhead) similar to OSA.\nExample: 1.5TB object created with 6 hosts (4+2) layout with RAID-6 SPBM\n","link":"https://blogs.hareeshkg.bid/post/vsan-esa-spbm-and-compoment-placements/","section":"post","tags":[],"title":"vSAN-ESA SPBM and Compoment placements"},{"body":"I have already briefly talked about vSAN-ESA’s Log Structured File System(LFS) in my previous blog, it advised to go over it once before you continue reading here.\nThis blog talks about the overview and architecture of LFS and how erasure coding is benefited when LFS was introduced into vSAN ESA architecture.\nHow LFS works? LFS in vSAN-ESA is responsible to handle all the incoming I/Os(Reads and Write) and acknowledge as fast as possible to the guest.\nWhen there is an incoming write IO from the guest , LFS ingests this write on to the “Performance leg” and sends the acknowledgment back to the guest immediately.\nIf the incoming IO request is a read, then depending on where the block resides the IO acknowledgement is either directly done at the read-cache layer or performance leg, if the IO didn’t exist in the read-cache or performance leg, it is fetched from the “Capacity leg” along with some additional blocks as part of prefetch, which gets cached and send acknowledgement to the guest.\nWhen sufficient amount of data fills into the “Performance Leg”, data will start to be moved to the “Capacity Leg” . To understand what is a “Performance Leg” and a “Capacity Leg”, please continue reading..\nHow does an Object look like in vSAN_ESA? In vSAN-ESA we will have a Concatenated Object(forest) at the top level which will have a minimum of two tree underneath the CONCAT Tree which accounts for Performance and Capacity Legs.\nThe number of capacity legs under the Concat can vary depending on the size of the object, I will cover how component placement works for vSAN-ESA in a different Blog.\nPerformance Leg? The performance leg is a very small partition of any given object in vSAN ESA. The performance leg comprises of a Durable Log and a Metadata log section.\nThe performance leg cannot be more than 255GB in size per component and the number of mirrors(Components) depends on the storage policy assigned to it.\nIn the above example we see that storage policy assigned is RAID-6 (*see Capacity Section) so performance leg also needs to sustain two hosts failure or two component mirror failure, hence a three way mirrored layout is assigned to this performance leg section.\nThis performance leg comprises of two section a Durable log section and a metadata section.\nWrite I/Os coming from the guest VMs is intercepted by LFS, they get Coalesced within the host memory for a very short duration (few Micro Seconds) and is written to this Durable log.\nThe I/Os written to the durable log consists of both metadata writes and Data writes. Soon after an IO is completed on the Durable log it is acknowledged to the guest for completion. When durable log has enough data to perform one large block full stripe write it moves this data to the Capacity Leg.( A full stripe write avoids additional write amplification for read modified writes, typically 4x amplification for RAID-5 and 6x amplification for RAID-6).\nFull stripe writes also reduce amount of data moving across the network because it is reducing all known over heads for partial read modified writes.\nDurable log also moves the metadata writes to the Metdata-Log and when metadata section is full in the performance leg it will also be moved to capacity leg eventually.\nB+ Tree? B+ tree is an efficient caching technique, it is used in vSAN-ESA to keep track/record of all the mappings ( lookup tables for logical to physical block address translation) for the data and metadata an efficiently in vSAN-ESA, these cache pages will also gets persisted eventually on to the Capacity Leg section.\nCapacity Leg? Every Object data will get persisted eventually on this Capacity leg, depending on the size of the object we will have one or more Capacity legs below the top level Concat tree. As already explained in the durable log section, capacity leg receives fulls stripe writes from the performance leg and the components are placed across different fault domains depending on the storage policy defined for an Object.\nHow Erasure Coding benefits from LFS? As you read thru the above sections you might have already got the idea that the introduction of LFS and performance leg concept eliminates the additional overheads for write penalties for using Either RAID-5 or RAID-6, because all write directly acknowledged at the performance leg to the guests and later moved to the capacity legs.\nThis means we get the very similar performance when using RAID-1/ RAID-5/ RAID-6 because I/Os are always written to the performance mirrors. We can get the best of both worlds where on one side we can use RAID-EC(Erasure Coding) to get space efficiency and Performance like RAID-1. This was not possible with vSAN-OSA we have to make a decision up front by either choosing RAID-1 with number of failures to tolerate if we need high performance or choose RAID-EC for space efficiency. Please read thru official VMware Blog on vSAN-ESA here.\n","link":"https://blogs.hareeshkg.bid/post/vsan-esa-log-structured-file-system/","section":"post","tags":[],"title":"vSAN ESA – Log Structured File System"},{"body":"vSAN-8 launched with a brand new storage architecture for vSAN called VMware vSAN Express Storage Architecture or ESA along side with a few enhancements to vSAN-OSA which is the Original Storage Architecture.\nUsers are given an option at the beginning of vSAN Cluster setup with vSphere-8 to choose either of the architecture, in this blog I explain what are the key features and enhancements of vSAN ESA and how it compares to OSA.\nHistory of vSAN-OSA! At the time of vSAN OSA launch back in the days during vSphere 5.5 we only had 1G/10G networks, SSDs were super expensive vs HDDs and CPU cores were very limited, hence this architecture primarily focused to cater a decent performance with available hardware during the time.\nWhen new features were added to vSAN-OSA with vSphere 6.x release such as support for All-Flash Technology, Erasure coding (RAID-5 / RAID-6), Deduplication \u0026amp; Compression, encryption etc we started to see some tradeoffs in terms of performance when using one or more features and the performance penalty was a little bit higher when running a stretched cluster between two sites with these features. The total available raw capacity was also less because every diskgroup has to have a dedicated Cache disk which doesn’t contribute to over all raw usable capacity.\nvSphere 7.x/vSAN 7.x brought in significant performance improvements when using Erasure Coding, Compression-Only feature,Data in Transit Encryption. These features did improve performance overall, however vSAN OSA could not scale very well due to inherent architectural limitations to exploit capabilities new generation storage devices (PMEM,OPTANE,MLC,TLC,QLC), new generation CPUs and faster network devices(25G,40G and 100G). Having said that OSA will continue to stay in the market along with ESA, infact the 600Gb Write Buffer for cache disk is improved to 1.6Gb, I will talk about this on a separate article.\nWhy vSAN ESA? vSAN-ESA is an alternative optimized architecture to exploit full potential of the very latest hardware (Storage/CPU/Network) to provide highly efficient, scalable, and performant HCI clusters by driving latencies for the modern application/workload lower and providing very high throughput and IOPS.\nOSA vs ESA\nKey features and Enhancements in vSAN ESA Each section explained below deserves its own blog, I will try to write about them one at a time in the coming days.. here are the key interesting features and enhancements of vSAN ESA\nvSAN ESA retains most of the key components part of the OSA’s architecture while adding new layers in between the existing vSAN layers. It introduces a new Log Structured File System (vSAN-LFS) which is patented. LFS helps to Ingest incoming write quickly with minimal overhead with help of Durable log, Metadata log and B-tree Caching, thus faster acknowledgement back to virtual machines when using an SPBM policy such as RAID-1, RAID-5 and RAID-6. Provide native vSAN-Snapshot feature, which provides consistent performance for VMs with snapshot and super fast (Consolidation/Deletion).\nProvides Compression by default to all vSAN objects which happens at the top of the stack, this reduces overall network bandwidth usage and IOPS to the storage devices and eliminates need of additional steps down the stack. This feature can be configured to be turned off on per object basis with use of Storage Policy Based Management (SPBM).\nSimilar to Compression feature on ESA, checksum is enabled **by default **at top of the stack and is not configurable.\nEncryption also works in similar fashion as compression feature which is at the top of the stack and can be enabled/disabled globally (Both On-Disk and over Network),reduces CPU cost and I/O amplification.\nprovide adaptive RAID-5 storage policy for objects at times when cluster size is down to just 3 nodes (OSA needs a minimum of 4 hosts),thus providing lower storage consumption (1.33x) vs RAID-1 (2x) per object.\nExisting Distributed Object Manager (DOM) is also enhanced to accomodate the new LFS layer, CPU performance optimizations, and also adds a new Network-Schedular which intelligently manages VM IO-traffic and resynchornization traffic using round trip latency poller in addition to overall bandwidth usage.\nExisting Log Structured Object Manager (LSOM) also had an overhaul on data structures and I/O Engine to provide extremely fast Single Tier performance (Yes! No more Disk Groups welcomes a Storage Pool concept), allowing all certified disks in a host to contribute to raw space, helps to claim devices automatically when new devices are added.\nCluster** L**evel Object Manager (CLOM) was also optimized for ESA to allows upto 27000 components per host which allows denser/more component placement vs OSA which had 9000 components per host limit. To work with the new LFS architecture, CLOM now creates a large CONCAT(concatenated) object at the top level and will have two break out legs underneath it called a Performance-Leg and a Capacity-Leg, where the performance leg accepts writes at the LFS layer and ack back to Guest, later this data from Performance-Leg is destaged back to Capacity legs.\nCluster Monitoring, Membership, and Directory Services (CMMDS) and Reliable Datagram Transport (RDT) also had significant enhancements to reduce intra cluster traffic over the network using compression techniques and flow control\nConclusion vSAN ESA is ideal HCI solution for all users who wants to get very low and reliable latency along with very high IO performance by exploiting latest new-gen hardware devices (CPU/Networking/Storage) to their fullest.\nvSAN OSA which is still very good for many cases and will continue to co-exist with vSAN-ESA. This initial release of vSAN 8 doesnt support cluster conversion/upgrade from vSAN OSA to vSAN ESA, storage vMotion from OSA to ESA cluster is the only available option.\nPlease read official VMware blog here to know more about vSAN-ESA.\n","link":"https://blogs.hareeshkg.bid/post/my-take-on-vsan-esa/","section":"post","tags":[],"title":"My take on vSAN-ESA"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-performance/","section":"categories","tags":null,"title":"VSAN Performance"},{"body":"vSAN 7.0 U3 delivers improved performance with optimisations made to the data path when using RAID-5/6 erasure coding very similar to what was added to vSAN 7.0 U2 Performance enhancements , vSAN 7.0 U3 offers caching capability to the incoming writes who’s workloads are serialised and have sustained high outstanding IOPS, this caching capability for writes reduces significant amount of write amplification by enforcing full stripe writes always in the backend.This feature compliments the feature which was introduced with vSAN 7.0 U2 where the reads for large block sequential IOPS were cached.\nTo give a perspective of how standard writes are handled with RAID 6 , we will have a total of 6 operations (Read data, read parity-1, read parity-2, write data, write parity-1 and write parity-2) for every IO which is received from the guest, i.e 1000 IOPS from the guest will result in 6000 IOPS at the backend in vSAN, these IOPS will also have additional network overhead as these have to go to the respective hosts where the components for the VMs are placed resulting in low throughput and high latency.\nStandard Write vs Strided Write When cluster is upgraded to vSAN 7.0 U3, we will see a significant performance improvements for VMs which are always doing large sequential Write IOPS, where the incoming write IOPS are cached and it waits long enough to see a possibility for a full stripe write, i.e IOPS are not immediately committed to the backend rather cached in cache pages and when it receives enough data to perform a full stripe write, it simply clears the entire stripe row and commits a full-stripe write also called a “Strided Write“, which is completed in just two steps. This reduction in the write amplification also reduces the number IOPS which have to traverse between the host where the VM is running and where the components are placed. This overall achieves a very efficient, low latency and high throughput for the guest VM.\nStrided write vs Standard write for RAID-5\nAbove image is a similar example for a RAID-5 object, when the VM issues IOs to the backend usually it takes 4 operations with a standard write, however when cached and written as a strided write its achieved with just two steps. Once again we see that write amplification is reduced significantly by reducing number of operation needed to complete the IO at storage and network layer.\nStrided write feature is always opportunistic and will generally work well with erasure coding R5/R6 for workloads with I/O sizes are sufficiently large enough to span across RAID row – 3MB for RAID-5, or 4MB for RAID-6. (e.g. 32 OIOs each with 128K I/O size) or write IO bursts – moderate OIO. (e.g. 8 OIO each with a 512KB I/O size). Having explained the feature “standard write” vs “strided write“, vSAN does not always receive full stripe writes. If incoming writes are not sequential, does not have a lot if outstanding I/O, or does not exceed the size of the stripe row, the more-efficient strided write method will not be used. This is dynamically determined by vSAN and such cases it will have to fall back to standard write .\nStrided writes are unable to be performed in a stretched cluster or 2-node where nested levels of resilience is used. In those topologies, the traditional method for writing data to the stripe is used instead.\nFor VMC Users VMConAWS consumers will benefit from these enhancements around strided write performance starting from version M16 which is equivalent to vSAN 7.0 U3.\n","link":"https://blogs.hareeshkg.bid/post/vsan-performance-70-u3-whats-new/","section":"post","tags":[],"title":"vSAN Performance 7.0 U3 Whats new?"},{"body":"In this blog I will go over some of the key benefits with vSAN 7.0U2. I got my hands on a 4 node VxRail Cluster, where I was able to run a few HCIBench tests to compare performance difference between vSAN 7.0 U1 and vSAN 7.0U2. I won’t be able to write a performance comparison between 6.7.x / 7.0x and vSAN 7.0 U1 as there is a white paper published by Dell-EMC for vSAN 7.0U1 see VxRAIL-7.0.100-Performance-White-Paper.\nSummarizing vSAN 7.0U1 overall provides 25% to 30% improvement on RANDOM/Sequential-reads, ~21% improvement for Sequential/Random Writes with RAID-5 storage policy which comes very close to performance numbers with RAID-1 and provides additional space savings of 25% on the vSAN datastore.\nShould you upgrade to VxRail 7.0.200(7.0U2) for performance? I got my hands on a 4Node VxRail Cluster where I was able to specifically run tests where we expect to see some improvement around RAID-5/Raid-6 for large sequential writes and CPU utilization tweaks which was announced with vSAN 7U2 see : https://blogs.vmware.com/virtualblocks/2021/03/09/vsan-7-update-2/.\nCluster-Config Four node VxRail (4 Hosts- Dell Inc. VxRail E560F -2x Intel(R) Xeon(R) Gold 5215 CPU @ 2.50GHz with memory of 256GB each) ,Controller : Avago (LSI Logic) Dell HBA330 Mini, 1DG on each host with drive Model : KPM5XMUG400G (373 GB) for Cache-Tier and 4 x KPM5XRUG1T92 (1788 GB) Capacity Tier, Deduplication and Compression enabled.\nPerformance Test Easy-run test for 256K Block size , 1 thread, sequential writes with 100% working-set,8VMs and 8VMDKs each. I ran the same tests repeatedly before on 7.0U1 by de-staging / clearing write buffer on all the hosts just to get a consistent baseline performance on VxRail 7.0.100/vSAN 7 U1. See HCIbench test result below :\n`Datastore = VxRail-Virtual-SAN-Datastore-1fa9ca3a-6672-4ffe-8150-e0e0c26eb537 JOB_NAME\t= job0 VMs\t= 8 IOPS\t= 3469.93 IO/S THROUGHPUT\t= 867.00 MB/s R_LATENCY\t= 0.00 ms W_LATENCY\t= 18.44 ms 95%tile_R_LAT\t= 0.00 ms 95%tile_W_LAT\t= 25.00 ms =============================`\nI later upgraded to VxRail 7.0.200/vSAN 7.0U2 ran the same workload once again, repeated the same tests multiple times similarly to get a consistent performance baseline, to my surprise I was able to achieve close to 50% boost in overall throughput/ IOPS and significantly lower latency. See HCIbench test result below : `Datastore = VxRail-Virtual-SAN-Datastore-1fa9ca3a-6672-4ffe-8150-e0e0c26eb537 JOB_NAME\t= job0 VMs\t= 8 IOPS\t= 5833.15 IO/S THROUGHPUT\t= 1458.00 MB/s R_LATENCY\t= 0.00 ms W_LATENCY\t= 10.97 ms 95%tile_R_LAT\t= 0.00 ms 95%tile_W_LAT\t= 18.00 ms =============================`\nPerf-Chart Comparison Before and After Upgrade to VxRail 7.0.200 I/o Flow Correlation BeforeUpgrade-Dom-Client BeforeUpgrade-Dom-Owner BeforeUpgrade-Dom-CompMgr AfterUpgrade-Dom-Client AfterUpgrade-Dom-Owner AfterUpgrade-Dom-CompMgr To view these advanced performance charts on the VC-Ui, you may choose to navigate to “performance for support view” , I actually generated these above charts using open-source grafana tool, however same data can be viewed within VC-UI.\nConclusion Although vSAN 7.0U1 provides better performance than the previous releases, I would highly encourage you to upgrade to vSAN 7.0U2/VxRail 7.0.200 or higher. With VxRail 7.0.200/vSAN7U2 you will :\nImprove the performance on RAID-5/6 workloads especially large Sequential Writes/reads by ~50% Reduces the CPU/network utilization thereby schedule and process more IOPS, push more IOPS over network. Get additional space savings when you compare space requirement between RAID-1 and RAID5/6 as this is more viable for certain workloads for the given perf-improvements. Significant performance improvements seen on AMD-EPYC CPU platforms with better NUMA awareness.\n","link":"https://blogs.hareeshkg.bid/post/vsan-70u2-performance-whats-new/","section":"post","tags":[],"title":"vSAN 7.0U2 Performance-Whats New?"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vxrail/","section":"categories","tags":null,"title":"VxRAIL"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-7-u1/","section":"categories","tags":null,"title":"VSAN 7 U1"},{"body":"vSAN 7 U1 brought in significant changes to its method of creating components especially for objects which are larger than 255GB. This change was brought in adapt to minimum slack space requirements which is significantly lower than all previous versions of vSAN.\nI highly encourage you to read thru blog articles “Understanding vsan Objects and Components” and “Understanding VM-Storage-Policies” as the basics about object/component placement are discussed in those specific blogs.\nLets first discuss the changes which were brought in with vSAN 7 U1 for Slack Space requirement and later understand how the component placement changes with vSAN 7 U1, further understand how this is helping in reducing Slack Space requirements.\nWhat is Slack Space or transient capacity? •Transient Capacity is generated when an object is reconfigured by VSAN •Re configuration could be user triggered (EMM/Policy Change) or VSAN triggered (Repairs/Rebalancing) •A policy change on a 255 GB object with the original policy FTT1, SW=1 and target policy FFT1, SW=2 will result in 510 GB worth of transient capacity. •This Becomes Challenging when the VM or Object sizes are very large example if you want to change a storage policy of VM/VMDK of 10TB from RAID-1 to RAID-5 we will need free space equivalent to ~20TB to perform this policy change with older vSAN Transient Capacity requirement.\nWhat changed in vSAN 7 U1 for transient capacity? On the current release vSAN 7 U1 and later, CLOM follows a different layout for object creation by having a “Concat Object” at the very top of the tree and creates multiple legs depending on the RAID policy that is chosen for the objects and ensures that each of these RAID-legs do not contain components larger than (255*2GB) in size. So when any policy changes are made\nCLOM does not have to resync the entire object in one shot. It starts by working on each RAID-leg, hence the resync that is triggered for each leg is no more than (255*RAID-Factor)GB, per leg. RAID-Factor is 2 in most of the cases.\nCLOM now will get the flexibility of working on each leg of the CONCAT independently, it will continue to create work-items to resync multiple legs at a time not more than (255*RAID-Factor)GB. If the cluster runs out of transient space all remaining resync work-items are throttled at CLOM, pending resync will be shown under “Scheduled Resync” in the UI, once transient space is recovered after completing on going resync, it will starts to process other work-items.\nNew UI in vSAN 7 U1 to shows if policy change is ”Pending/In progress/ Failed” state •REDUCED_AVAILABILITY_WITH_POLICY_PENDING •REDUCED_AVAILABILITY_WITH_POLICY_PENDING_FAILED •NON_HFT_RELATED_INCOMPLIANCE_WITH_POLICY_PENDING •NON_HFT_RELATED_INCOMPLIANCE_WITH_POLICY_PENDING_FAILED\nThis new method of component placement is helping us by reducing transient/slack space requirement for storage policy changes and for failure handling at the cluster level by working on smaller subset of components (One leg at a time).\n​**Reserved Capacity **based on deployment variables in vSAN 7 U1: •12 node cluster = ~18% •24 node cluster = ~14% •48 node cluster = ~12%\nUnderstanding Component Placement in vSAN 7 U1 and above As discussed in the section “What changed in vSAN 7 U1 for transient capacity?“, we do not see any difference for VMs and objects which are smaller than 255GB, this change policy applies to the objects that are larger than 255GB. If you are currently on vSAN 7.0 GA or lower release, you will need to perform something called as “Object format conversion” after completing the upgrade and On-Disk-Format upgrade, you will be provided with a warning on vSAN-Health plugin to validate if you have enough capacity in the cluster to convert a few objects to the newer layout.\nOFC-Warning on vSAN-Skyline-health\nLets now Look at different Object layout with different Storage Policy in place.\n50 GB object with FTT-1,RAID-1 and Stripe Width of 1 There is no difference in this example of a 50GB VMDK as mentioned in the earlier section, the change is only seen with Objects larger than 255GB.\n50GB Object with FTT-1,RAID-1 and Stripe Width of 3. 50GB,FTT=1,RAID-1,SW=3.png\n50GB VMDK with FTT-3,RAID-1 and Stripe Width of 1 50GB,FTT=3,RAID-1,SW=1.png\n500GB VMDK with FTT-1,RAID-1 and Stripe width of 1 500GB,FTT=1,RAID-1,SW=1.png\nIn this Example of a 500GB object we see that it created a CONCAT tree at the top, created two RAID-1 Legs, each RAID-1 Leg have two components each. So if there were to be a policy change in the further for this VMDK, CLOM will work with each leg individually, the policy change will first resync the first RAID-1 Tree (2552)GB to the new layout defined by the user, later moves to resync the second RAID-1 tree (2452)GB to the new layout, as mentioned earlier CLOM is allowed perform resync for multiple legs concurrently as long as cluster has enough transient space available.\n500GB VMDK with FTT-1,RAID-1 and Stripe width of 3 500GB,FTT=1,RAID-1,SW=3.png\nAs mentioned in the picture number CLOM has decided to create RAID-1 Trees with two RAID-0 sets on each leg. First set of RAID-0 group have four components each with 85 GB and the second set of RAID-0 group with ~81.6GB. Each RAID-1 is less than or equal to (255*2=510GB).\n500GB VMDK with FTT-3,RAID-1 and Stripe width of 1 500GB,FTT=3,RAID-1,SW=1.png\n500GB VMDK with FTT-1,RAID-5 and Stripe width of 1 500GB_FTT-1_RAID-5_SW-1to4.png\nFTT-1 Objects with RAID-5, stripe width definition between 1 and 4 will always create four components on each tree. We will see a two component increase for every odd number of stripe width meaning SW-5 and SW-6 will have 6 components on each RAID Tree, SW=11 and SW=12, will have 12 components on each RAID Tree.\n500GB VMDK with FTT-2,RAID-6 and Stripe width of 1 500GB_FTT-2_RAID-6_SW-1to6.png\nFTT-2 Objects with RAID-6, stripe width definition between 1 and 6 will always create six components on each tree. We will see a two component increase for every odd number of stripe width meaning SW-7 and SW-8 will have 8 components on each RAID Tree, SW=11 and SW=12, will have 12 components on each RAID Tree.\nCreating pictorial depiction for all types of storage policy gets really complex. I have attached this file “Sample_storage_policy.txt”, which contains object layout for various different storage policies if you want to understand how placements will look for different kind of storage policies and stripe width.\nSample_storage_policy.txt Download We also have an official blog written around storage policies for VMC-AWS on vSAN, take a look at “Improved Storage Policy Capacity Management“, want to learn about other interesting features on vSAN-7 U1 see LINK.\n","link":"https://blogs.hareeshkg.bid/post/vsan-7-u1-component-placement/","section":"post","tags":[],"title":"vSAN 7 U1 Component Placement"},{"body":"vSAN Scrubber is a background operation at DOM owner. Scrubber performs detection (scrub) and fixing (recover) of checksum and IO errors.\nMostly Checksum errors and I/O errors which are seen in vSAN environment is due to medium errors against the physical capacity disks where the data is persisted. These errors don’t surface until we try to read the object from both mirrors (in case of raid-1 objects). We might always be performing reads on a single mirror especially in case of Stretched cluster where site affinity is set to either preferred site or secondary site to avoid cross site traffic.\nMaintenance tasks, hardware failures which results in a resync task , will trigger reads against the secondary/surviving mirrors which may discover and report checksum errors or I/O failures due to bad sectors backing the object’s component and resync would get stuck.\nvSAN 7.0U1 and previous release had the advanced scrubber value “VSAN.ObjectScrubsPerYear” set to **1 per year **against each objects, this setting is now changed to 6 per year on the latest vSAN 7.0U1c and later releases. This means that vSAN Scrubber will scrub every objects once every two months to make sure all the affected components with unreadable blocks, incorrect checksum are relocated to different sectors/disks by rebuilding from neighboring components/mirrors. See ESXI 7.01c release notes for additional information.\n![](/images/vsan-scrubber-changes-70u1c-scrubber-blog-dec-1.png) We can also look at some statistics against certain VMDK if necessary to know if there were any checksum errors against a specific object and ETA for scrubber completion (Generally used by VMware support**)\n`[root@localhost] esxcli vsan debug object list -u 19433d5f-9dee-9f58-9b67-ecf4bbec65d8 Object UUID: 19433d5f-9dee-9f58-9b67-ecf4bbec65d8 Version: 13 Health: healthy Owner: is-tse-d155.vsensei.local Size: 50.00 GB Used: 8.90 GB Policy: objectVersion: 13 hostFailuresToTolerate: 1 SCSN: 92 CSN: 94\nConfiguration:\nRAID_1 Component: 19433d5f-e068-605c-faf8-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 53687091200 (50.00GB), Disk UUID: 52de9507-470b-8a3c-1916-3dc6c696bc5f, Disk Name: naa.5002538c4044d881:2 Votes: 1, Capacity Used(B): 4827643904 (4.50GB), Physical Capacity Used(B): 4777312256 (4.45GB), Host Name: is-tse-d156.vsensei.local Component: 19433d5f-6436-635c-631f-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 53687091200 (50.00GB), Disk UUID: 52deab75-cff7-87a7-1273-5a505995c264, Disk Name: naa.5002538c4044d87f:2 Votes: 1, Capacity Used(B): 4827643904 (4.50GB), Physical Capacity Used(B): 4777312256 (4.45GB), Host Name: is-tse-d157.vsensei.local Witness: 19433d5f-7f0b-645c-55c6-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 525a9779-affc-46ec-5f59-70a0b6ae28d2, Disk Name: naa.5002538c4044d6ad:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d155.vsensei.local Type: vdisk Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/14433d5f-ce0e-0a0c-8d9f-ecf4bbec65d8/VC-50_12.vmdk (Exists) Group UUID: 14433d5f-ce0e-0a0c-8d9f-ecf4bbec65d8 Directory Name: N/A\n[root@localhost] vsish -e get /vmkModules/vsan/dom/owners/19433d5f-9dee-9f58-9b67-ecf4bbec65d8**/scrubStats DOM Owner Scrub Stats ** Total bytes scrubbed during the last round:56224055296 Total bytes allocated for this object:9554624512 Total size of all components including parity if any:107374182400 Number of rounds of scrubbing completed:0 Number of scrub ops issued:10 Total number of LSEs/checksum errors detected:0 Total number of faked checksum errors detected:0 Total number of OOB LSEs/checksum errors detected:0 Total number of OOB faked checksum errors detected:0 Total number of LSEs/checksum errors recovered:0 Total number of faked checksum errors recovered:0 Total number of OOB LSEs/checksum errors recovered:0\nTotal number of OOB faked checksum errors recovered:0 Total number of LSEs/checksum errors that failed to recover:0 Total number of faked checksum errors that failed to recover:0 Total number of OOB LSEs/checksum errors that failed to recover:0 Total number of OOB faked checksum errors that failed to recover:0 Seconds since the last round of scrubbing started:11501811 Total bytes scrubbed so far in current round:0 Initial estimation of current scrub ETA (seconds):36432237 Current estimation of current scrub ETA (seconds):558692354 Maximum estimation of current scrub ETA (seconds):3071998125 Maximum estimation of last scrub ETA round (seconds):0`\nAlso take look at the new blog post on [Advanced-Cross-vCenter Migration ](/post/advanced-cross-vcenter-vmotion/)feature which is included with vSphere 7.0U1c. ","link":"https://blogs.hareeshkg.bid/post/vsan-scrubber-changes-70u1c/","section":"post","tags":[],"title":"vSAN Scrubber changes 7.0U1c"},{"body":"VMware had initially launched a fling version called “Cross vCenter Workload Migration Utility” which was used by a lot of VMware administrators/customers to migrate workloads/VMs between two vCenters which are managed individually (not under the same SSO domains). This advanced “Cross-vCenter vMotion”, a.k.a. XVM GUI tool is now natively built into vSphere 7.0U1c which allows bulk transfer of VMs between vCenter instances without the need of any additional fling/plugin installation. XVM does not depend on linked mode, which means that the vcenter servers can be in a separate SSO domains and can be used for migration between vCenter servers on-premise (VC to VC) and in the cloud (VC to VMC on any cloud platforms).\nXVM/Cross-vCenter vMotion leverages vMotion APIs available in all versions since vCenter 6.0U3.\nMigrate VMs to vCs on premise and in the cloud from any vC 6.0U3 or higher Migrate VMs to the cloud which allows customers to bulk migrate VMs to VMC-AWS and other VMC cloud providers. How does this work On any vSphere environment running vCenter 7.0U1c or above, you may simply Right-Click on a cluster or a host which will give you an option to “Import VMs” which you may choose to perform bulk migration of VMs from any previous version of vCenter environment to this environment. Similarly if you want to migrate individual VMs from a vSphere 7.0 U1c environment to a different vCenter not in the same SSO domain, you will see a new option when we choose the option migrate against VMs.\n- ![](/images/advanced-cross-vcenter-vmotion-1-XVM.png) Choose VM(s) to migrate, select migrate\nFind the new option for “cross vCenter Server export”\nchoose New vCenter if destination vCenter credentials were not saved already\ntype the credentials for destination VC\naccept certificate for the destination VC\nconnectivity should be established between the two VC\nchoose the destination hosts/cluster\nselect datstore/SPBM\noptional\nSelect destionation/target network\nreview/complete wizard\n- ![](/images/advanced-cross-vcenter-vmotion-XVM-import-1.png) *Importing VMs straight from Target/Destination vCenter *\nSelect or add new Source VC\neasy to bulk import multiple VMs\nchoose target host/cluster\nchoose datastore/SPBM\noptional\nDestination VM Network\nFinish\n","link":"https://blogs.hareeshkg.bid/post/advanced-cross-vcenter-vmotion/","section":"post","tags":[],"title":"Advanced Cross-vCenter vMotion"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/general-vsphere/","section":"categories","tags":null,"title":"General-VSphere"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsphere-7.0u1c/","section":"categories","tags":null,"title":"Vsphere 7.0U1c"},{"body":"vSAN hosts reboots when compared to regular ESXi hosts generally take longer due to its log structured system to capture and track all the modifications occurring at the write buffer / Cache Disks, vSAN uses an in-memory table to track changes to the blocks, LBA tables, bitmaps etc. This in memory table is always lost when a vSAN host is rebooted. At the time of boot vSAN takes a long time to recover these entries from each of the disk groups to reconstruct the in-memory table, the time spent to perform this recovery generally depends on the size of LLOG/PLOG that needs to be recovered from each of the disk groups. vSAN 7.0U1 / vSAN 7 Update 1 performs a “save and restore” ​workflow to rebuild vSAN in-memory metadata allowing “Faster vSAN Host Restarts” or Fastboot.\nWhen a host is placed in maintenance mode, the workflow will be initialized to quiesce/ serialize all the PLOG data and perform a check on the SSD Cache disk to see if enough free space is available to save this data, once enough space is identified it will write the PLOG data to this location on the SSD cache disk.\nWhen host is rebooted, log recovery process will de-serialize PLOG data log from the same location and loads it into the in-memory table a lot faster by avoiding manual log recovery procedure which is generally seen in the previous releases. Hence we see ~5x faster reboot times as it does not spend a lot of time to rebuild the PLOG in memory table at time time of boot. This fastboot feature does not work if the host had crashed with a PSOD, disk failure or if new drives were added to the DG when the host was placed in maintenance mode.\nvSAN 7.0 U1- Fastboot workflow\nSuccessful FASTBOOT Log Entries zcat /var/log/boot.gz | egrep -i \u0026quot;Fast | fastboot\u0026quot; | less\n2020-09-20T07:34:13.255Z cpu1:263673)LSOMCommon: LSOMFbOpen:355: disk: 5269015b-c76b-638a-1a14-906e569ef428 LSOM Fastboot Context 0x45006a7b1a40, start offset: 1048576, end offset: 214695936 2020-09-20T07:34:13.255Z cpu1:263673)LSOMCommon: LSOMFbInitCtx:161: disk: 5269015b-c76b-638a-1a14-906e569ef428 Initialized fastboot de-serialization context 0x45006a7aa960, cprGenNr 179 2020-09-20T07:34:13.298Z cpu3:263673)PLOG: PLOGFbFastRecoverDevice:1384: disk: 5269015b-c76b-638a-1a14-906e569ef428 Fastboot completed successfully 2020-09-20T07:34:13.496Z cpu3:263675)LSOMCommon: LSOMFbOpen:355: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 LSOM Fastboot Context 0x4500543b6ae0, start offset: 1048576, end offset: 214695936 2020-09-20T07:34:13.496Z cpu3:263675)LSOMCommon: LSOMFbInitCtx:161: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Initialized fastboot de-serialization context 0x4500543c6d00, cprGenNr 2066 2020-09-20T07:34:13.562Z cpu3:263675)PLOG: PLOGFbFastRecoverDevice:1384: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Fastboot completed successfully 2020-09-20T07:34:13.562Z cpu3:263675)LSOMCommon: LSOMFbFreeCtx:229: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Freeing fastboot context: 0x4500543c6d00 2020-09-20T07:34:13.562Z cpu3:263675)LSOMCommon: LSOMFbClose:407: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 LSOM Fastboot Context 0x4500543b6ae0 close 2020-09-20T07:34:13.562Z cpu3:263675)PLOG: PLOGRecoverDeviceLogsDispatch:5198: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Completed fast recovery successfully 2020-09-20T07:34:13.562Z cpu3:263675)LSOMCommon: LSOMFbOpen:355: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 LSOM Fastboot Context 0x4500543b6ae0, start offset: 1048576, end offset: 214695936 2020-09-20T07:34:13.564Z cpu3:263675)LSOMCommon: LSOMFbClose:407: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 LSOM Fastboot Context 0x4500543b6ae0 close\nFastboot failure after a PSOD zcat /var/log/ | egrep -i \u0026quot;Fast | fastboot\u0026quot; | less\n2020-09-20T07:07:56.751Z cpu3:263673)LSOMCommon: LSOMFbOpen:355: disk: 5269015b-c76b-638a-1a14-906e569ef428 LSOM Fastboot Context 0x45006a7a3880, start offset: 1048576, end offset: 214695936 2020-09-20T07:07:56.752Z cpu2:263673)LSOMCommon: LSOMFbInitCtx:125: disk: 5269015b-c76b-638a-1a14-906e569ef428 Invalid fastboot header for super block 2020-09-20T07:07:56.752Z cpu2:263673)LSOMCommon: LSOMFbClose:407: disk: 5269015b-c76b-638a-1a14-906e569ef428 LSOM Fastboot Context 0x45006a7a3880 close 2020-09-20T07:07:56.752Z cpu2:263673)PLOG: PLOGFbFastRecoverDevice:1326: disk: 5269015b-c76b-638a-1a14-906e569ef428 Failed to init the fastboot context 2020-09-20T07:07:56.752Z cpu2:263673)PLOG: PLOGRecoverDeviceLogsDispatch:5204: disk: 5269015b-c76b-638a-1a14-906e569ef428 Fast recovery failed : Failure 2020-09-20T07:07:56.752Z cpu2:263673)LSOMCommon: LSOMFbOpen:355: disk: 5269015b-c76b-638a-1a14-906e569ef428 LSOM Fastboot Context 0x45006a7a3880, start offset: 1048576, end offset: 214695936 2020-09-20T07:07:56.835Z cpu3:263675)PLOG: PLOGFbFastRecoverDevice:1326: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Failed to init the fastboot context 2020-09-20T07:07:56.835Z cpu3:263675)PLOG: PLOGRecoverDeviceLogsDispatch:5204: disk: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Fast recovery failed : Failure 1\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34; style=\u0026#34;\u0026#34;\u0026gt;**\u0026lt;u\u0026gt;When FASTBOOT \u0026lt;/u\u0026gt;**\u0026lt;/span\u0026gt;**\u0026lt;span style=\u0026#34;text-decoration: underline;\u0026#34;\u0026gt;\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34;\u0026gt;falls back to manual log recovery :\u0026lt;/span\u0026gt;\u0026lt;/span\u0026gt;** 2 32020-09-20T07:07:57.212Z cpu0:263202)LSOMCommon: SSDLOG_SetBlkMap:283: device: 5269015b-c76b-638a-1a14-906e569ef428 BlkMap state changed from SSDLOG_BLKMAP_INITIALIZED to SSDLOG_BLKMAP_TRACKING, segNo:0, type:1 42020-09-20T07:07:57.268Z cpu3:263201)PLOG: PLOGAllocDevState:3771: 528cade0-9b4c-a8cc-ee5c-3d2bbec524f1: PLOG state num WB buckets: 4, bucket size 2629886976 5**\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34;\u0026gt;2020-09-20T07:07:57.269Z cpu3:263201)PLOG: PLOGHandleLogEntry:375: Recovering SSD state for MD 52d0a4e6-4b5d-1d6c-7b8a-7ccf2b62f85d 62020-09-20T07:07:57.269Z cpu3:263201)PLOG: PLOGAllocDevState:3771: 52d0a4e6-4b5d-1d6c-7b8a-7ccf2b62f85d: PLOG state num WB buckets: 4, bucket size 2629886976 72020-09-20T07:07:57.775Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 1500 of ~73785 (2%) log blocks. 0s so far, ~24s left.\u0026lt;/span\u0026gt;** 8. 9. 102020-09-20T07:07:59.575Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 4500/~73785, 6%, 2s/~36s. 112020-09-20T07:07:59.627Z cpu1:263202)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5269015b-c76b-638a-1a14-906e569ef428 Recovery progress: 4500 of ~7229 (62%) log blocks. 2s so far, ~2s left. 122020-09-20T07:07:59.627Z cpu1:263202)vSAN Recovery: 5269015b-c76b-638a-1a14-906e569ef428 4500/~7229, 62%, 2s/~2s. 132020-09-20T07:08:00.629Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 6000 of ~73785 (8%) log blocks. 3s so far, ~40s left. 142020-09-20T07:08:00.629Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 6000/~73785, 8%, 3s/~40s. 152020-09-20T07:08:00.691Z cpu3:263202)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5269015b-c76b-638a-1a14-906e569ef428 Recovery progress: 6000 of ~7229 (82%) log blocks. 3s so far, ~3s left. 16**\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34;\u0026gt;2020-09-20T07:08:01.540Z cpu2:263673)PLOG: PLOGRecDisp:999: PLOG recovery complete 5269015b-c76b-638a-1a14-906e569ef428:Processed 110218 entries (unmap 0), Took 4348 ms\u0026lt;/span\u0026gt;** 172020-09-20T07:08:01.542Z cpu2:263673)Destroyed VSAN Slab PLOGRecovSlab_0x450040015860 (maxCount=0 failCount=0) 18**\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34;\u0026gt;2020-09-20T07:08:01.542Z cpu2:263673)PLOG: PLOGRecDisp:1010: Recovery for 5269015b-c76b-638a-1a14-906e569ef428 completed with Success\u0026lt;/span\u0026gt;** 19. 20. 212020-09-20T07:08:34.733Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 67500 of ~73785 (91%) log blocks. 37s so far, ~37s left. 222020-09-20T07:08:34.733Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 67500/~73785, 91%, 37s/~37s. 232020-09-20T07:08:35.504Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 69000 of ~73785 (93%) log blocks. 38s so far, ~38s left. 242020-09-20T07:08:35.504Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 69000/~73785, 93%, 38s/~38s. 252020-09-20T07:08:36.302Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 70500 of ~73785 (95%) log blocks. 39s so far, ~39s left. 262020-09-20T07:08:36.302Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 70500/~73785, 95%, 39s/~39s. 272020-09-20T07:08:37.085Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 72000 of ~73785 (97%) log blocks. 39s so far, ~39s left. 282020-09-20T07:08:37.085Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 72000/~73785, 97%, 39s/~39s. 292020-09-20T07:08:37.892Z cpu2:263201)LSOMCommon: SSDLOGLogEnumProgress:2200: device: 5278ef79-0866-7f0a-42a8-bb7a2477f843 Recovery progress: 73500 of ~73785 (99%) log blocks. 40s so far, ~0s left. 302020-09-20T07:08:37.892Z cpu2:263201)vSAN Recovery: 5278ef79-0866-7f0a-42a8-bb7a2477f843 73500/~73785, 99%, 40s/~0s. 312020-09-20T07:08:38.096Z cpu2:263201)LSOMCommon: SSDLOGEnumLog:2385: PLOG: Total Time: 40828818 us, Read Time: 40422590 us, Process Time: 4455184 us, numReads: 73170 32 33**\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34;\u0026gt;2020-09-20T07:08:38.142Z cpu3:263675)PLOG: PLOGRecDisp:999: PLOG recovery complete 5278ef79-0866-7f0a-42a8-bb7a2477f843:Processed 881024 entries (unmap 21), Took 40874 ms\u0026lt;/span\u0026gt;** 342020-09-20T07:08:38.144Z cpu3:263675)Destroyed VSAN Slab PLOGRecovSlab_0x45004001a880 (maxCount=0 failCount=0) 35**\u0026lt;span class=\u0026#34;has-inline-color has-vivid-cyan-blue-color\u0026#34;\u0026gt;2020-09-20T07:08:38.144Z cpu3:263675)PLOG: PLOGRecDisp:1010: Recovery for 5278ef79-0866-7f0a-42a8-bb7a2477f843 completed with Success\u0026lt;/span\u0026gt;** Read about other features with vSAN 7.0 U1 / vSAN 7 Update1 [here](/vsan-7-u1-whats-new) or on vmware-official blogs [here](https://blogs.vmware.com/virtualblocks/2020/09/15/whats-new-in-vsan-7-update-1/) ","link":"https://blogs.hareeshkg.bid/post/vsan-7-update-1-fastboot-feature-walkthrough/","section":"post","tags":[],"title":"vSAN 7 Update 1 FastBoot – Feature Walkthrough"},{"body":"What is Durability Component?, Lets consider a scenario of a three node vSAN cluster, where one of the host is placed in maintenance mode. All the vSAN objects will now be accessible with “Reduced Redundancy” mode which means that any failures that are encountered on the surviving two nodes will lead to data unavailability and sometimes data loss situation even with all recovery efforts from a support perspective.\nWhenever there is one or more hosts placed in maintenance mode with ensure accessibility, vSAN 7 Update 1 introduces a method to write new changes by spawning a new component called “Durability Component“. Since new changes for the objects that are being written to this “Durability Component” if we were to encounter a failure on one of surviving host, exiting maintenance mode on the host which has stale components will sync the changes with the Durability Component and becomes an active component again and the Durability Component is automatically cleaned up by the CLOM service on that node . This helps ensure the durability of the changed data in the event that the one host holding the updated object replica failed during this maintenance window\nIn the below illustration we have 3 out of 5 hosts in maintenance andVM-win10 has a component which is marked as absent on host “10.78.190.90” which is currently in maintenance mode. Soon after we lose the last surviving mirror, CLOM service will automatically spawn a new Durability Component, in this case on host “10.78.180.113” even though this violates the vSAN default storage policy (RAID_1 / FTT-1) considering there is a witness component already placed on node “10.78.180.113”. This is to provide additional protection if there was a failure permanent failure on Host “10.78.181.100” and we can exit maintenance mode on host “10.78.190.90” the Absent-stale component will sync up with Durability Component and bring the object to an accessible state without a data loss situation.\nNote : If there was a permanent failure on host “10.78.180.113” while host “10.78.190.90” was still in maintenance, the object will go to inaccessible state since we lost the witness component and the durability component at the same time, there is no chance of lossless recovery. However this feature still increases the chances of recoverability if “10.78.181.100” was lost.\nDurability component spawned when we have 3/5 hosts in maintenance mode Ensure accessibility for an FTT-1/Raid-1 Object.\nThat concludes this blog article on ” Enhanced Durability Maintenance Mode–** Feature-Walkthrough**“. If you liked this blog article feel free to share with your colleagues and friends.\nContinue reading about other features on vSAN 7.0 U1 / vSAN 7 Update 1 here\n","link":"https://blogs.hareeshkg.bid/post/enhanced-durability-maintenance-mode-feature-walkthrough/","section":"post","tags":[],"title":"Enhanced Durability Maintenance Mode – Feature Walkthrough"},{"body":"**HCI Mesh ** is one of the new feature that is introduced with vSAN 7 Update 1 release. HCI Mesh provides disaggregation of compute and storage by allowing vSAN customers to share vSAN datastores between two or more clusters. vSAN administrators can setup a relationship between two to five clusters and allow these clusters to borrow/share vSAN datastore capacity.\nWe can achieve better compute and storage utilization across the datacenter and avoid unnecessary purchase of hardware per cluster. Example : VM hosted on cluster-1 can have all of its disks and namespace provisioned from a different cluster within the same datacenter, allows vSphere administrators to do a regular vMotion (not SvMotion) between one or more clusters which are part of this **HCI Mesh ** topology. This feature uses native vSAN VMKernel adapter and existing RDT protocol to communicate in the mesh topology with other cluster vSAN nodes which is simplistic and efficient.\nTo enable **HCI Mesh ** topology between two or more clusters, GO to Cluster \u0026gt; Configure \u0026gt; vSAN \u0026gt; Datastore Sharing, and click on “Mount remote datastore.” The UI will allow administrators to mount vSAN datastores from other cluster within the vSphere Datacenter where the current cluster is residing, next it triggers a compatibility check wizard before initiating the mounting process.\nTerminologies Used with **HCI Mesh **: Client Cluster – The local cluster where we are trying to mount a remote vSAN datastore Server Cluster- The remote cluster whose datastore is being mounted to the local cluster.\nHCIMesh-Design\nCompatibility Checks: •The remote datastore type is vSAN – This check prevents non vSAN datastore to be listed as available remote datastore for mount. •**Server and client clusters are from the same datacenter **– Remote vSAN cluster has to be on the same vSphere Datacenter in the vCenter server. •**The selected server cluster is remote **– Selected server cluster is not same as the local cluster. •**Cluster configuration is supported **– Prevents Stretched and ROBO remote vSAN cluster to be listed as compatible •**vSAN format version supports remote vSAN – Server cluster is on vSAN 7 Update 1 with latest On-Disk-Format version. •[client vSAN datastore] is mounting from no more than 5 server clusters **– Local/Client vSAN cluster cannot mount more than 5 remote vSAN datastore. •No more than 5 client clusters are mounting [server vSAN datastore] vSAN’s datastore– Server cluster vSAN datastore is not mounted to more than 5 remote clusters •The remote vSAN datastore can provision objects with its default policy – Validates by checking if VMs can be provisioned with default datastore policy •Server cluster is healthy – Server vSAN cluster vSAN skyline health check passes all major tests. •Server and client cluster have no connectivity issues– vSAN VMKernel adapters between vSAN hosts on both server and client clusters can communicate over RDT protocol. •Latency between client and server hosts is below 5ms– Just like local vSAN cluster health check the network latency between the server and client (Local and Remote) vSAN cluster members should be less than 5ms to ensure that we dont have any major performance compromise while we disaggregate compute and storage between two clusters.\nLimitations •HCI Mesh feature is part of vSAN Enterprise or higher licensing •vSAN Direct Configuration– vSAN Direct datastores cannot mount a remote vSAN datastore or be mounted to a client cluster •vSAN datastores using Data-in-transit encryption cannot be presented to client clusters •**Stretched Clusters and 2-Node clusters **vSAN datastores cannot be mounted to a client cluster. •vSAN cluster services like File Services, iSCSI, or CNS based FCD workloads can only be presented to VMs running on Local vSAN Datastore and must not be presented to VMs on Remote vSAN clusters. •Air-gapped vSAN networks, or clusters using multiple vSAN vmkernel ports are not supported with HCI Mesh. •All objects for any virtual machine should be presented from either local or remote vSAN datastore, mixing placements across two or more vSAN datastores is not supported.\n- ![](/images/hci-mesh-feature-walkthrough-Datastore-Sharing-1-1024x437.png) Datastore Sharing-Wizard\nSelect a Datastore to mount\nCompatibility fails for a ROBO vSAN Datastore while mounting\nWhen we choose a valid standard vSAN datastore to mount check passes\nAfter successfully mounting a Remote vSAN datastore\nCapacity View Post mounting\nRemote DS perf metric view\nVM-Placement view when components are placed on Remote vSAN Datastore\nVM-Objects VIew only shows VM object placement within the local vSAN Datastore and doesnot show placement for VMs running off Remote clusters\nShared-Datastore-view-Cluster level\nThat concludes this article on “**VMware HCI Mesh ** – Feature Walkthrough” , make sure to read my other blogs on vSAN 7 U1 feature walk-through/guides. If you liked this blog article feel free to share with your colleagues and friends.\nRead about other features with vSAN 7.0 U1 / vSAN 7 Update1 here or on official VMware blogs here\n","link":"https://blogs.hareeshkg.bid/post/hci-mesh-feature-walkthrough/","section":"post","tags":[],"title":"HCI Mesh™ -Feature Walkthrough"},{"body":"vSAN 7 Update 1 now gives us high efficiency to this two node “Shared Witness” topology by allowing vSAN administrators to use a single shared witness for up to 64 two node vSAN clusters and hence reduces witness server/appliance footprint/resource usage in the central datacenter. The new UI for the witness appliance under Cluster-\u0026gt;Monitor-\u0026gt;vSAN-\u0026gt;Two Node Clusters.\nTwo-Node vSAN ROBO cluster configuration was introduced some time back with vSAN 6.2 release couple of years back, this architecture was used primarily in large number of small branch offices, retail sites, IOT etc. All two node vSAN cluster configuration always needed a dedicated witness appliance for each of its deployment until vSAN 7. This means that there was a large footprint of witness appliances in the central site which hosts witnesses for all of its retail two node ROBO sites.\nCertain Design and availability constraints might require multiple shared witness dedicated to a group of two node clusters, since losing a shared witness host will have a higher magnitude of vulnerability when it is server a large number of two node clusters and all the objects that shares the witness node will go to reduced availability state. It is recommended to deploy additional witness nodes in the centralized location to quickly replace the witness if the primary witness server fails so that all objects can be re-protected immediately by replacing the witness server. Shared witness appliance/server will require additional memory resources ~12GB for up to 30 two node clusters and needs additional if the size grows higher.\nConfigure Shared witness To configure a “shared witness” we need to make sure that the witness server which is deployed has a minimum memory allocation of 12GB if not the witness VM can be powered-off, increase the memory allocation and power it back on. vSAN Engineering team is planning to include options to change the compute resources for shared witness during witness-OVA/OVF deployment.\nIn the following Example (screenshots) the witness appliance is already configured for ROBO-1 Cluster, the same witness server will be configured to serve as witness node for cluster : ROBO-2.\n- ![](/images/shared-witness-feature-walkthrough-robo-1-1024x288.png) Verify witness memory allocation\nReview current witness configuration\nStart ROBO Cluster configuration\nenable necessary services for new cluster\nClaim Disks for data nodes\nReview and Complete Wizard\nReview Witness Config once again\nThat concludes this article on Shared Witness on vSAN 7 Update 1 , make sure to read my other blogs on vSAN 7 U1 feature walk-through/guides. If you liked this blog feel free to share with your colleagues and friends.\nRead about other features with vSAN 7.0 U1 / vSAN 7 Update1 here or on vmware-official blogs here\n","link":"https://blogs.hareeshkg.bid/post/shared-witness-feature-walkthrough/","section":"post","tags":[],"title":"Shared Witness – Feature walkthrough"},{"body":"This blog article is a feature walkthrough on “vSAN 7 Update1 IOInsight” and understand how this can help us to approach/investigate performance related issues on VMs running on vSAN.\nIOinsight has been available in the VMware-Flings community market place for quite sometime. vSAN engineering team has now included IOInsight natively with vSAN Performance services with this latest release of vSAN7 Update 1. This provide deep dive performance metrics and insights on individual VM and VMDK level, so that this data can be used to mitigate performance related issues on specific VMs and VMDKs.\nWhy do you need vSAN IOInsight? When you look at the cluster performance data i.e (Cluster=\u0026gt;Monitor=\u0026gt;vSAN=\u0026gt;Performance=\u0026gt; VM/BACKEND) you may see that the overall cluster latency might show higher than generic threshold (25ms and above), after reviewing various performance charts at host level and individual VM level you might see that there may be one or more VMs which are actual culprits which is impacting overall performance of your vSAN cluster.\nFurther you will need additional performance metrics for those specific VMs to understand the IO Size Distribution, Histogram, Read \u0026amp; Write IO Ratio, Sequential \u0026amp; Random IO Ratio,Sequential \u0026amp; Random Throughput etc. These metrics are not available on the vSAN native performance charts, for administrators to share the required data for the VM owner these graphs will help the root cause the problem.\nIOInsight helps to introspect specific VM’s performance metrics at a granular levelby tracking all performance metrics for every IO add the vSCSI layer. All above mentioned performance metrics will be captured and can be exported as a CSV file and PNG/JPEG for data analytics.\nHow to start vSAN IOInsight? To start an IOInsight monitoring session go to Cluster=\u0026gt;Monitor=\u0026gt;vSAN=\u0026gt;Performance , select IOInsight and click on “NEW INSTANCE” , it would allow you monitor and collect metrics for all the VMs in the cluster or you can choose specific target VMs to start collecting IOInsight perf data, next you will need to choose run time in mins/hours. Once an instance is started you can pre-emptively stop the collection before the runtime expires or wait for it to auto end collection.\nReview Analyze results After you start an IOInsight instance, you may choose to wait for the defined time period to expire or stop the instance at any time. You can also view live performance metrics while the instance is running. Once the test finishes or stopped, you can perf data for the instance or export the data as CSV/SVG images as a zip file and view them offline. See below screenshots to know how we export data and review perf data as CSV and as image files.\n- ![](/images/ioinsight-vsan-70-u1-feature-walkthrough-Export_link_IOINSIGHT-1-1024x426.png) How to export\nchoosing matrix to export\nexported data – view\nIOPS\nThroughput\nSequential \u0026amp; Random Throughput\nSequential \u0026amp; Random IO Ratio\n4K Aligned \u0026amp; Unaligned IO Ratio\nRead \u0026amp; Write IO Ratio\nIO Size Distribution\nIO Latency Distribution\nThat concludes this article on vSAN IOInsight , make sure to read my other blogs on vSAN 7 U1 feature walk-through/guides. If you liked this blog feel free to share with your colleagues and friends.\nRead about other features with vSAN 7.0 U1 / vSAN 7 Update1 here or on vmware-official blogs here\n","link":"https://blogs.hareeshkg.bid/post/ioinsight-vsan-70-u1-feature-walkthrough/","section":"post","tags":[],"title":"IOInsight vSAN 7.0 U1 – Feature walkthrough"},{"body":"vSAN 7.0 U1/ vSAN 7 Update1 is one of the biggest release for vSAN in last couple of years, there is a long list of new features that are added/released with this version. This blog article talks briefly on all the important features for vSAN7 Update1 / vSAN 7.0 U1. I will be covering all the features exclusively on newer dedicated posts for each of the features.\n-1 **VMware HCI Mesh ** -vSAN 7.0 U1\n**VMware HCI Mesh ** (Datastore sharing) AKA compute-storage disaggregation is a feature that allows VI administrators to Use capacity across independent vSAN clusters (upto 5) so that we can separate compute from storage whenever needed to better use resources (Compute and storage) across clusters. There is a list of pre-requisites to enable this feature, make sure check out the blogpost on vSAN-HCI Mesh Feature Walkthrough on vSAN7 Update1 / vSAN 7.0 U1.\n-2 Shared Witness (ROBO Clusters)-vSAN 7.0 U1 Shared witness feature allows vSAN administrators to use a common Witness Appliance (Witness node) for multiple two-node ROBO vSAN clusters, this means we alleviate the requirement to use dedicated witness node for each two-node robo clusters. A single witness appliance can be used for upto 64 two node ROBO clusters. Make sure to checkout the blogpost –Shared Witness Feature Walkthrough on vSAN7 Update1 / vSAN 7.0 U1\n-3 vSAN Capacity Optimization with variable reserves -vSAN 7.0 U1 With all release version of vSAN until vSAN 7 GA, we always had to retain a free capacity or slack space between 25% to 30% of total vSAN datastore capacity to accommodate host-failure/Disk Group/Disk failure rebuilds. With this latest release for vSAN 7 Update 1 we no longer need to have 25% of reserved capacity for Slack Space. The vSAN Capacity UI clearly indicates how much slack space is required for host rebuilds. To avail this new capacity optimization feature you will need to perform an object level re-layout if the cluster was upgraded from a previous release. Check out vSAN Capacity Optimization vSAN 7 Update 1 blogpost for additional information.\n-4 vSAN Compression-Only -vSAN 7.0 U1 We also have a compression only mode for space savings with vSAN 7 Update1, its no longer required for administrators to enable both Deduplication and Compression together for space savings. Enabling compression only feature gives more flexibility and performance benefits, check out detailed blog Compression Only Feature Walkthrough on vSAN7 Update1 / vSAN 7.0 U1.\n** -5 vSAN I/O Insight** -vSAN 7.0 U1 IoInsight was available as an ovf appliance on the VMware Flings website for quite sometime, with this release of vSAN 7 Update 1 I/O Insight has been integrated natively on vSAN performance monitoring service, this will be used for collecting detailed performance metrics of individual VMs at a granule level .This help us approach performance related issues better and helps us in resolving performance issues on a vSAN cluster. Check out vSAN 7 Update1 I/O Insight Feature Walkthrough on vSAN7 Update1 / vSAN 7.0 U1 post for additional information.\n-6 Enhanced durability during Maintenance Mode Whenever we had to performance host level maintenance on a vSAN cluster, individual hosts are required to be placed in maintenance mode and most frequently used method is ensure data accessibility. During this maintenance the components on the host in maintenance are marked absent and surviving components on other hosts continue to service I/O if the storage policy in use was FTT-1, there is a change that any of the surviving host to fail and we may encounter data loss situation if we could not recover from the second failure.\nThis release of vSAN 7 U1, allows a host to entire maintenance mode with ensure accessibility by re-directing all new writes for the components placed in the host to different host, if there was a subsequent failure on any of the surviving nodes we can exit maintenance on the host which was placed in maintenance and allow the stale components to resync with newly created delta components and make the objects accessible. Check out post Enhanced Durability Maintenance Mode – Feature Walkthrough on vSAN7 Update1 / vSAN 7.0 U1.\n-7 vSAN Direct Datastore vSAN Direct is a new type of vSAN datastore which basically presents all the local storage devices on the hosts to cloud native storage applications hosts on the cluster thru Kubernetes control plane in vSphere . vSAN Direct Configuration optimizes the data path and minimizes capacity consumption for share nothing architectures and allows you to bypass vSAN data services that are already built into the application layer of modern stateful services.\n**-8 Enhancements to vSAN File Services ** vSAN File services was introduced with vSAN 7 release, however not all features were supported with this release. With the version of vSAN 7.0 U1 / vSAN 7 Update1, we now have support for Kerberos authentication for File shares and added SMB support for windows file shares and VDI home profile support natively on vSAN.\n-9 vSAN Data-in-Transit Encryption vSAN Data-in-transit encryption feature ensures encryption over the wire similar to vMotion encryption it uses **FIPS 140-2 **validated VMware VMkernel Cryptographic module, this encrypts data and metadata flow over the network and doesnot require a KMS server to enable this feature. DiT encryption can be enabled thru vSAN Data Services after upgrading to vSAN 7 Update1, also to note that if DiT encryption is enabled we cannot enable HCI-Mesh. Check out vSAN Capacity Optimization vSAN7 Update1 / vSAN 7.0 U1 blogpost for additional information. This feature also supports Data in transit encryption for vSAN file services and SMB services, DiT Encryption however is not supported when HCI-Meshing is enabled between clusters.\n-10 Faster reboot on vSAN Nodes vSAN 7 Update 1 uses a “save and restore” workflow /mechanism to rebuild vSAN in-memory metadata during node reboots or when a disk group is unmounted/remounted. The in-memory metadata table in the previous versions was read serially from the disk group metadata partition every time a vSAN node was rebooted, in this version vSAN hosts dumps the current in-memory metadata to a location on Cache disks of each of the disk groups and when the host is rebooted, it just reads from the dump location, the reboot time when compared to previous version is almost 5-10x faster. Check out blogpost Fast-Reboot vSAN7 Update1 / vSAN 7.0 U1 for additional information.\n–11 Routed vSAN Topologies vSAN Stretched and Robo clusters no longer requires you to add additional routes on the hosts individually for vSAN data node and witness appliance communication, the vSAN VMKernel adapters gateway configurations can be overridden on the UI while creating/configuring vSAN vmkernel adapters.\nThat concludes this article on vSAN 7.0 U1 /vSAN 7 Update1 overview, make sure to read my other blogs on individual feature deep dives. If you liked this blog article feel free to share with your colleagues and friends.\nRead about other features with vSAN 7.0 U1 / vSAN 7 Update1 on vmware-official blogs here.\n","link":"https://blogs.hareeshkg.bid/post/technical-overview-vsan-7-update-1/","section":"post","tags":[],"title":"Technical Overview – vSAN 7 Update 1"},{"body":"In this blog article we will look at the vSAN 7 U1 Compression-Only feature ,its benefits and how it fares with other space saving features on the previous releases of vSAN.\nI have been asked by a lot vSAN users about vSAN only allowing us to enable Deduplication and Compression together as feature in all the releases since vSAN 6.x. Finally VMware made this announcement for vSAN compression only mode is here with vSAN 7 Update1 we can totally eliminate the additional overheads for Deduplication when using Compression-Only yet get good space savings. Let dive straight into key difference between Deduplication \u0026amp; Compression vs Compression-Only.\n**Deduplication and Compression****Compression-Only**• Cache tier SSD has to perform both DD\u0026C at the same time.For every Disk Group with DD\u0026C we only have a single elevator (Destage thread) that constraints the rate at which we are destaging writes from cache tier• Cache tier SSD only has to account for compression while doing De-stage and we have an elevator (Destage-thread) for every disk in the capacity tier, depending on the workload we may see **almost 1.5X performance (DD\u0026C)**• A **capacity tier drive failure will fail the entire Disk Group**, which means entire Disk Group will needs to be recreated.• A **capacity tier drive failure will not unmount the entire Disk Group**, affected disk can be replaced and re-added into the Disk Group. • **Huge amount of resync** will be triggered for a drive failure as entire Disk Group will be unmounted and the resync that is triggered will be slightly larger than the Diskgroup itself as the data is un-deduped and un-compressed. Once the data is resync to the Diskgroup, the data will be DD\u0026C on the destination Disk group.• **Resync is triggered only for the failed drive**, we reach compliant state sooner than Dd\u0026C feature,the amount of data inflight when a capacity drive fails is far lesser than DD\u0026C. *Difference between DD\u0026C and compression-only* Compression only feature will be very realistic for customers who wants to have a decent level of space efficiency and who cannot afford the performance impact of DD\u0026amp;C. Large block sequential write performance, OLTP Database VMs are expected to see better performance with compression only vs DD\u0026amp;C.\nCompression-Only see the difference- UI ![](/images/vsan-7-u1-compression-only-feature-walkthrough-vsan_comp_only_screenshot_oldVer.png) Old version of UI for vSAN-7 and below VS all new vSAN 7 U1\nThat concludes this blog article on ” vSAN 7 U1 Compression-only Feature-Walkthrough“, I will try to post a new blog specific about performance improvements with vSAN compression only and how it compares with both Dedupe and compression. If you liked this blog article feel free to share with your colleagues and friends.\nContinue reading about other features on vSAN 7.0 U1 / vSAN 7 Update 1 here\n","link":"https://blogs.hareeshkg.bid/post/vsan-7-u1-compression-only-feature-walkthrough/","section":"post","tags":[],"title":"vSAN 7 U1 Compression-Only – Feature Walkthrough"},{"body":"In this blog you will see how we upgrade your VxRail cluster to version 7.x. The pre-requisites and approach to upgrade to VxRAIL 7..x has quite few changes when compared to previous versions/releases.\nNote: VxRail upgrades is usually done through VxRail support over a remote session or thru onsite VxRail Systems Engineer from DELL-EMC or through Dell-EMC partners. Ensure to read thru the DELL-EMC Solve procedure before attempting an upgrade or contact DELL-EMC support for assistance.\nConsideration before Upgrade Download and Read through the official VxRail-7.0.00-Release notes and Support-Matrix (DELL-EMC portal login required) Internet/online upgrade from the VxRAIL plugin on the VC is not supported for this release. All Quanta based VxRail hardware are not supported for upgrade to version 7.x VMware Update Manager or the new VMware Life Cycle Manager is not supported during upgrade or post upgrade. Upgrade from VxRail version 4.7.411 and all versions below 4.5.211 to version 7.0.000 is not supported. Certain VxRail environments using nVIDIA GPUs are not supported for upgrade to 7.0.000 VxRail -7.0 upgrade Pre-Requisites The following pre-requisites applies to most of the VxRail appliances, however as discussed earlier, you will need to access DELL-EMC Solve Desktop to generate all the steps for your upgrade to version 7.0.\nFirst step is to run the VxVerify scripts from KB:536801(portal auth required), this will run multiple scripts on the VxRailManager, VxRail Nodes, vCenter server and the platoform services controller**. (** if they are not external to VxRAIL appliance), if there were any errors during this pre-check verify scripts you will need to address them run the script again to verify before starting the upgrade. In this below example, I was performing an upgrade of VxRail version 4.7.410 on E560F VxRail four node cluster to version 7.0.000, pre-upgrade script completed with few warnings, which were not a blocker for upgrading this cluster. Make sure you have all required credentials with administrator and root privileges to all management vms check Monitor \u0026gt; Issues and Alarms \u0026gt; All issues and similarly Check vSAN health plugin once again. Ensure all of them are in a healthy state and address any issues before attempting upgrade. Download the Upgrade Composite bundle from the recommended link in the DELL-EMC solve procedure, in my case it was: VXRAIL_COMPOSITE-7.0.000-26451670_for_4.7.x.zip which was around 6.9GB. Take snapshots of all the management VMs: VxRail Manager VM, VMware vCenter Server Appliance,VMware vCenter Server Platform Services Controller,VMware vRealize Log Insight VM and ESRS VE. If the Vcenter was external it must be already upgraded to version 7.0 GA and snapshot is not required if its external VC. Start the Local upgrade from VxRail plugin on the HTML-5 vCenter server UI and the bundle downloaded using DELL-EMC Solve Desktop. The upgrade may take around 6 hours to complete. Post Upgrade Summary\nAfter Upgrade is complete,ensure you delete all the snapshots on the management appliances after monitoring the cluster for a day or two, running VMs on snapshot will lead to performance issues and un-necessary space consumption on vSAN datastore. Dont miss out to read about the biggest release of vSAN 7 Update 1 , a lot of features announced with this release check out article on vSAN 7 Update 1-Technical Overview\nNote: the upgrade procedure/steps to VxRail 7.0.100 (vSAN 7 U1) and later releases will remain same, make sure to read thru respective release notes and support matrix\n","link":"https://blogs.hareeshkg.bid/post/how-to-upgrade-to-vxrail-70/","section":"post","tags":[],"title":"How to upgrade to VxRail™  7.0"},{"body":"SRM 8.3 and vSphere Replication 8.3 bring in long awaited features,UI changes, security and performance improvements. In this blog we shall be going over some of the features and enhancements which you may consider to upgrade from the previous releases to this version as soon as possible as these features helps to reduce a lot of manual work and adds value to your every day task. I have not covered steps to deploy VR and SRM appliances as I have already written a blog post for Install-Configure-migrate-SRM-VR-8.2, you may follow same steps for deployment and configuration.\nWhat’s new in SRM 8.3? ![](/images/srm-83-vsphere-replication-83-walk-through-SRMicon.png) - Automated VM Protection - vSphere-Client UI Enhancements - New Import/Export Tool ​ - vRops Mgmt pack Enhancements - vVol integration and auto protection - vRO Automation Enhancements - Security What’s new in vSphere Replication 8.3? ![](/images/srm-83-vsphere-replication-83-walk-through-VR-icon.png) - Integrated Disk Resizing - ​Performance Improvements - New Import/Export Tool - ​vSphere-Client UI Enhancements - ​New vROps Mgmt Pack Automated VM Protection with SRM 8.3 Array-Based SRM 8.3 automatically protects new virtual machines created/moved to datastores which are part of an array-based replicated luns. There is no requirement to add the VM to the protection group and recovery plan. This feature also applies for vVols which are protected with new SRM 8.3, administrators may simply change the storage policy for the VM to a vVol policy which is protected will SRM, VMs are automatically added to Protection group and Recovery Plan.\nIf there are two datastores which are in the same protection group, if you move a VM from one to another with SvMotion, VM will be still auto-protected. Watch the Below GIF example where I storage vMotion a VM from vSAN Datastore to EMC-Storage backed datastore which was part of protection group and recoverplan, VM is automatically protected.\nAutoProtection (Click the GIF to popup)\nNote:* Unprotection is not covered with this feature if an administrator deletes a VM which was protected with SRM, manual clean up is required on PG/RP. ****Auto unprotection doesn’t work if destination datastore is not protected or if it is part of a different Protection Group\nAutomatic vDisk resizing in VR Re-sizing virtual machine disk that is currently protected by vSphere Replication (VR) does not require any additional steps at the Destination datastore or hosts. vDisk(s) at the destination site will automatically be resized, hence saves us from following the KB: https://kb.vmware.com/s/article/2042790 which is still required for older vSphere release 6.x and before. Lets take a look at this feature in the below GIF demo.\nIn this Demo the VM “Photon-vSAN” which is currently running on a vSAN Datastore will be resized will actively replicating to another vSAN datastore at the secondary site.\nVR-Automatic-Disk-Expansion (Click the GIF to popup)\nImport-Export Tool for vSphere Replication vSphere Replication 8.3 brings in this new feature to export or backup all vSphere Replication configuration data. When we encounter situations where vSphere Replication appliance needs to be re-deployed on either of the sites, we now can reduce VR Re-Deployment time by importing configuration from a backup file.\nVMware currently supports this using a Java based Import-Export tool (vr-import-export-tool-8.3.0-15928802.zip) similar to import-export tool which was available for SRM in previous releases, it is available for download HERE. This can be used from any machine which has connectivity to the VC and VR appliances at both sites and requires JAVA 1.8 or above to be installed all instructions to use this tool is available HERE . In the below screenshots you can see how the tool is used to export VR configuration.\nUsing vr-import-export-tool to export config\nExported file should available on the same folder as the tool\nUI Enhancements in VR 8.3 VR 8.3 UI will now all users to export replication details of all VMs as CSV file , which includes SRM Protection Group information,has a new RPO column,replica disk usage and RPO violation.\nSample Replication Export data from VR UI\nOther Features in SRM and VR 8.3 VR 8.3 also include performance improvements packed into it, VR 8.3 completes initial sync significantly fast by not comparing checksums for empty disks and also skips read operation while performing initial sync and just replicates the data to destination folder. Export import tool which was already available for SRM, the new tool ​can preserve all mappings not configured to “use site-level” during import. Export/backup can be part of the recover plan. SRM 8.3 now supports Array Based vVol replication. VMs part of vVol Datatsores which are being protected by array based replication are automatically protected using the storage policy which has definitions for vm protection. vROPS Management pack for SRM 8.3 unconfigured VMs that are part of a PG, and lists VMs associated with PGs vROPS Management pack for VR 8.3 shows ​Top 10 VMs with RPO violations and ​replication lag time. vSphere Replication data usage and consumption on vSAN Datastore has a dedicated view within vSAN Capacity Consumption Reporting, highly encourage you to go thru the official VMware Blog that I had posted vsan-7-capacity-reporting-enhancements. Similar to VR UI enhancement , SRM also allows administrators to export all datagrids to Excel/CSV, option for ​Show/Hide columns to all datagrids. Note: Upgrade to SRM/VR 8.x ASAP as older SRM/VR 6.5.x requires Adobe flash and flash will no longer be distributed/supported as of Dec 31st 2020.\n","link":"https://blogs.hareeshkg.bid/post/srm-83-vsphere-replication-83-walk-through/","section":"post","tags":[],"title":"SRM 8.3 \u0026 vSphere Replication 8.3 walk through"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/srm-vspherereplicaiton/","section":"categories","tags":null,"title":"SRM-VsphereReplicaiton"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan6.7/","section":"categories","tags":null,"title":"VSAN6.7"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/whats-new/","section":"categories","tags":null,"title":"Whats New?"},{"body":"With introduction of vSAN 6.7 U3 / vSphere 6.7 U3 we now have native support for SCSI-3 Persistent reservation on vSAN objects. This means that we no longer need vSAN iSCSI targets for creating application level clustering services like Windows Server Failover Clusters (WSFC) which needs to share a disk in order to determine ownership across the application cluster.\nThis new feature allows for VMs running WSFC to use a shared VMDK on a vSAN datastore by creating SCSI controllers in physical Mode and then create and share vmdks using independent-persistent presentation mode. This drastically reduces the time in deployment and configuration on WSFC when compared to previous vSAN releases which requires us to configure thru vSAN iSCSI targets and also comes with its overhead .\nConfiguring vSAN native WSFC Create a new virtual SCSI Controller using the “physical” option set for the SCSI Bus Sharing setting. This is the virtual SCSI controller that will be used by the shared VMDK. Add new SCSI Controller\nFor the given VMDK that will be shared, assign it to the virtual SCSI Controller created above, and ensure the disk mode is set to “Independent – Persistent” Go to other VMs and add them using “Add Existing Disk option” on which you would want to add the shared drives, make sure you select your new SCSI controller which was created and set the drives to independent persistent mode. See below GIF for complete setup\nOnce the drives are shared you can go to the guest and claim the drives , following Demo-GIF only shows that the drives can be claimed and used within guest Recommended Practices Once the drives are shared between the two VMs , it is highly recommended that you setup VM to VM anti-affinity rule so that both clustered VMs are not moved by DRS automatically to the same host to avoid simultaneous failures. For Stretched cluster implementation :Double-check the VM to VM affinity rules and VM to Host affinity rules and set them up correctly. Its important to note that the Objects cannot be owned by two hosts at the same time , which means that if we create shared disks on VM-1 and power it on on host-1 the ownership stays with host-1 for the respective shared vmdk and secondary VM-2 which shares the disk with VM-1 will send all its IOPS over the vSAN VMkernel traffic over to the object owner to commit its IOPS and this might use good amount of vSAN bandwidth between the sites when implemented in stretched clusters . If all the heavy lifting is done on the primary object owner we will see greater response times for the IOPS and when site fails all shared disk object ownership will move to the secondary VM Depending on the practicality and requirements you can chose to setup storage policy to tolerate failures across and within sites “PFTT/SFTT” Example Object owner comparison [root@host1:~] esxcli vsan debug object list --vm-name=Node1-WSFC Object listed below are truncated namespace,vmswap and first OS disk objects are not listed , showing only shared disks from VM-1 Object UUID: 9e1beb5d-87b9-1d24-5a3d-ecf4bbec65d8 Version: 10 Health: healthy Owner: host2.vsensei.com Size: 10.00 GB Used: 0.11 GB Policy: stripeWidth: 1 cacheReservation: 0 CSN: 3 forceProvisioning: 0 spbmProfileName: vSAN Default Storage Policy proportionalCapacity: 0 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad hostFailuresToTolerate: 1 spbmProfileGenerationNumber: 0 Configuration: RAID_1 Component: 9e1beb5d-a78d-9524-231c-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 52d97584-3471-9660-934a-e2cafd267313, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 62914560 (0.06GB), Physical Capacity Used(B): 58720256 (0.05GB), Host Name: host1.vsensei.com Component: 9e1beb5d-16f9-9624-787d-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 52ece953-87e3-0fe4-8ab8-cbdd374aa0f3, Disk Name: naa.5002538c4044d6ae:2 Votes: 1, Capacity Used(B): 62914560 (0.06GB), Physical Capacity Used(B): 58720256 (0.05GB), Host Name: host3.vsensei.com Witness: 9e1beb5d-c8ca-9724-8c77-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 525452e3-e6c0-884a-a6ea-edc03570a490, Disk Name: naa.5002538c4044d6ab:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host1.vsensei.comType: vdiskPath: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/071aeb5d-21b3-664d-7ae0-ecf4bbec6050/Node1-WSFC_1.vmdk (Exists)Group UUID: 071aeb5d-21b3-664d-7ae0-ecf4bbec6050Directory Name: N/A Object UUID: 8f1deb5d-abd5-f134-85ec-ecf4bbec65d8Version: 10Health: healthyOwner: host2.vsensei.comSize: 40.00 GBUsed: 0.20 GBPolicy:stripeWidth: 1cacheReservation: 0CSN: 3forceProvisioning: 0spbmProfileName: vSAN Default Storage PolicyproportionalCapacity: 0spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5badhostFailuresToTolerate: 1spbmProfileGenerationNumber: 0Configuration: RAID_1 Component: 8f1deb5d-6bba-8535-5e36-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 42949672960 (40.00GB), Disk UUID: 5225900e-2911-3548-1c57-3fb38e948a00, Disk Name: naa.5002538c4044d881:2 Votes: 1, Capacity Used(B): 109051904 (0.10GB), Physical Capacity Used(B): 104857600 (0.10GB), Host Name: host1.vsensei.com Component: 8f1deb5d-833a-8735-aa62-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 42949672960 (40.00GB), Disk UUID: 52cf328a-533c-5162-12ef-bff7ac01d103, Disk Name: naa.5002538c4044d6ad:2 Votes: 1, Capacity Used(B): 109051904 (0.10GB), Physical Capacity Used(B): 104857600 (0.10GB), Host Name: host1.vsensei.com Witness: 8f1deb5d-fb06-8835-4f8c-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 52208d4c-7f55-578a-cf89-21b65b58e3a3, Disk Name: naa.5002538c4044d87f:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host3.vsensei.comType: vdiskPath: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/071aeb5d-21b3-664d-7ae0-ecf4bbec6050/Node1-WSFC_2.vmdk (Exists)Group UUID: 071aeb5d-21b3-664d-7ae0-ecf4bbec6050Directory Name: N/A\n``` **When we look at debug object list output was looked from VM-2 which is sharing the disk from VM1 , it does not show the shared disk because the attached disks belong to a different namespace UUID and also for a fact that we cant have two object owners for the same object the ownership always stays with the first VM-1.**[root@host2:~] esxcli vsan debug object list --vm-name=Node2-WSFC**Object UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050**Version: 10Health: healthy**Owner: host3.vsensei.com**Size: 255.00 GBUsed: 0.79 GBPolicy:cacheReservation: 0spbmProfileGenerationNumber: 0stripeWidth: 1forceProvisioning: 0spbmProfileName: vSAN Default Storage PolicyhostFailuresToTolerate: 1spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5badproportionalCapacity: [0, 100]CSN: 3Configuration:` RAID_1 Component: f119eb5d-3031-090c-559b-ecf4bbec6050 Component State: ACTIVE, Address Space(B): 273804165120 (255.00GB), Disk UUID: 52cf328a-533c-5162-12ef-bff7ac01d103, Disk Name: naa.5002538c4044d6ad:2 Votes: 1, Capacity Used(B): 423624704 (0.39GB), Physical Capacity Used(B): 419430400 (0.39GB), Host Name: host1.vsensei.com Component: f119eb5d-5169-0a0c-4104-ecf4bbec6050 Component State: ACTIVE, Address Space(B): 273804165120 (255.00GB), Disk UUID: 52ece953-87e3-0fe4-8ab8-cbdd374aa0f3, Disk Name: naa.5002538c4044d6ae:2 Votes: 1, Capacity Used(B): 423624704 (0.39GB), Physical Capacity Used(B): 419430400 (0.39GB), Host Name: host3.vsensei.com Witness: f119eb5d-512f-0b0c-8869-ecf4bbec6050 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 52d97584-3471-9660-934a-e2cafd267313, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host1.vsensei.com`Type: vmnamespace*** Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/Node2-WSFC ***(Exists)Group UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050Directory Name: Node2-WSFC **Object UUID: f319eb5d-8ed8-cf3b-3494-ecf4bbec6050**Version: 10Health: healthy**Owner: host3.vsensei.com**Size: 40.00 GBUsed: 19.39 GBPolicy:cacheReservation: 0spbmProfileGenerationNumber: 0stripeWidth: 1forceProvisioning: 0spbmProfileName: vSAN Default Storage PolicyhostFailuresToTolerate: 1spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5badproportionalCapacity: 0CSN: 3Configuration:` RAID_1 Component: f319eb5d-fe40-843c-ec0f-ecf4bbec6050 Component State: ACTIVE, Address Space(B): 42949672960 (40.00GB), Disk UUID: 525452e3-e6c0-884a-a6ea-edc03570a490, Disk Name: naa.5002538c4044d6ab:2 Votes: 1, Capacity Used(B): 10510925824 (9.79GB), Physical Capacity Used(B): 10406068224 (9.69GB), Host Name: host1.vsensei.com Component: f319eb5d-5bdf-853c-ff7b-ecf4bbec6050 Component State: ACTIVE, Address Space(B): 42949672960 (40.00GB), Disk UUID: 52d97584-3471-9660-934a-e2cafd267313, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 10510925824 (9.79GB), Physical Capacity Used(B): 10406068224 (9.69GB), Host Name: host1.vsensei.com Witness: f319eb5d-7c0e-873c-ccb2-ecf4bbec6050 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 52208d4c-7f55-578a-cf89-21b65b58e3a3, Disk Name: naa.5002538c4044d87f:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host3.vsensei.com`Type: vdisk**Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/f119eb5d-f922-730b-4de1-ecf4bbec6050/Node2-WSFC.vmdk** (Exists)Group UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050Directory Name: N/A**Object UUID: fa3aeb5d-f5eb-6ef0-f766-ecf4bbec91a8**Version: 10Health: healthy**Owner: host3.vsensei.com**Size: 8.00 GBUsed: 0.01 GBPolicy:cacheReservation: 0spbmProfileGenerationNumber: 0stripeWidth: 1forceProvisioning: 1spbmProfileName: vSAN Default Storage PolicyhostFailuresToTolerate: 1spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5badproportionalCapacity: 0CSN: 3Configuration:` RAID_1 Component: fa3aeb5d-b08c-e5f0-ee09-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 8589934592 (8.00GB), Disk UUID: 52cf328a-533c-5162-12ef-bff7ac01d103, Disk Name: naa.5002538c4044d6ad:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host1.vsensei.com Component: fa3aeb5d-e2a4-e6f0-99f1-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 8589934592 (8.00GB), Disk UUID: 52d97584-3471-9660-934a-e2cafd267313, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host1.vsensei.com Witness: fa3aeb5d-ce5d-e7f0-b3a2-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 52ece953-87e3-0fe4-8ab8-cbdd374aa0f3, Disk Name: naa.5002538c4044d6ae:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: host3.vsensei.com`Type: vmswap** Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/f119eb5d-f922-730b-4de1-ecf4bbec6050/Node2-WSFC-3f77264a.vswp** (Exists)Group UUID: f119eb5d-f922-730b-4de1-ecf4bbec6050Directory Name: N/A ``` For Additional information on Microsoft SQL clustering setup with SCSI-3 PR read ***[VMware-Blogs-native-sql-cluster-setup-vSAN](https://blogs.vmware.com/virtualblocks/2019/03/04/native-sql-server-cluster-support-on-vsan/)*** and[ ***VMware-Blogs-******Native-WSFC-vSAN-Stretched-Clusters***](https://blogs.vmware.com/virtualblocks/2019/10/09/wsfc-stretched-clusters/) ","link":"https://blogs.hareeshkg.bid/post/wsfc-with-native-shared-disks-on-vsan/","section":"post","tags":[],"title":"WSFC with Native Shared Disks on vSAN"},{"body":"In this guide we will be looking at how we can use different storage policy on vSAN datastore to fine tune performance for the virtual machines running on a vSAN Datastore . We will look at different tools which can be run from the guest level and also few tests run with HCIBench .\nBefore you start reading thru this blog I highly encourage you to check out “Achieve-Best-Performance-on-vSAN” to get any understanding vSAN performance benchmark tool called HCIBench.\nPlease ensure to also go thru the post “understanding-vm-storage-policies” to get a hold of storage policy base management and different rule-sets / parameters within storage policy .\nIn this test case , I got my hands on one of a three Node Dell-Servers running vSAN 6.7 U1 All-FLASH with Dedupe and compression enabled , each host has 1 disk group with 1 SSD cache tier drive and two capacity tier drive and I ran two different BenchMarking tool “HCIBench” from VMware on this cluster and another in-guest level Disk Benchmark tool called “Crystal DiskMark” against individual VMDKs to compare performance with two different storage policy .\nHardware Specs - server model and BIOS version Dell Inc. PowerEdge R630 | BIOS: 2.7.1\n#[Rackmount] [id:35865] DELL PowerEdge R630 #vcgLink = 'https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=server\u0026amp;productid=35865' #picture = 'https://www.google.com/search?q=DELL%20PowerEdge%20R630\u0026amp;source=lnms\u0026amp;tbm=isch\u0026amp;sa=X' - CPU type Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz # Link: 'http://www.google.com/#q=E5-2630+ark.intel.com' Number of packages (pCPU):2 Number of cores (total):16 Hyperthreading state: 3 -\u0026gt; enabled\n- CPU microcode revision Current Revision:0x0000003d\n- Network Interface Card 2X - Intel Corporation Ethernet Controller 10 Gigabit X540-AT2\n- Host Bus Adapter vmhba driver driver version model\nvmhba0 lsi_mr3 7.703.18.00-1OEM.650.0.0.4598673 Avago (LSI) PERC H730 Mini\n- HBA queue settings PERC H730 Mini Integrated : 891\nController HCL : https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vsanio\u0026amp;productid=34859\nDrives used for vSAN esxcli storage core device list |grep -i 'Devfs Path|vendor|Model|Revision|Raid|size' | egrep -v Queue |sed 'N;N;N;N;N;s/\\n//g'\nSize: 190782 Devfs Path: /vmfs/devices/disks/naa.55cd2e414d9b2945 Model: INTEL SSDSC2BA20 Revision: DL2B RAID Level: NA Size: 915715 Devfs Path: /vmfs/devices/disks/naa.5002538c4044d6ab Model: MZ7KM960HAHP00D3 Revision: GB52 RAID Level: NA Size: 915715 Devfs Path: /vmfs/devices/disks/naa.5002538c4044d6ad Model: MZ7KM960HAHP00D3 Revision: GB52 RAID Level: NA Cache Tier HCL : https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd\u0026amp;productid=39831 - \u0026quot;CLASS-E 30,000-100,000 writes per second\u0026quot; Capacity Tier HCL : https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd\u0026amp;productid=39915 - \u0026quot;CLASS D: 20,000-30,000 writes per second\u0026quot; RUN-**HCIBench ** From the above specs its clear that we can only define the storage policy with fault tolerance method as RAID-1 with stripe width either as SW=1 or SW=2 going higher stripe width is point less as we only have two capacity tier drives and one Disk group . Hence I did run different tests with SW=1 and SW=2 . Here are the result comparison . Note: The following illustrations is used only to show difference between two storage policy , do not use them as expected benchmark results in your environment as these results are purely environment dependent .\nTest result on Storage Policy Definition as failures to tolerate -1 and stripe-width=1 gives with the workload definition “6 VMs, 8-vDisks, 80% working set , 4K Block Size, 70% read% and 80% random packets , number of threads as 4” is 31111 IOPS and throughput of 121MB/s .\nThe same workload was next run with Storage Policy Definition as failures to tolerate -1 and stripe-width=2 , we see that this policy was able deliver 59017 IOPS and throughput of 230MB/s this is almost two times the IOPS and throughput achieved previously by only increasing the stripe width from one to two .\nPlease click thru the following Images to see test results :\nRunning Guest Level BenchMark! To validate if we are actually benefited from higher stripe width , we may also run some guest level performance benchmark tests . For this test I chose to use/run Crystal-DiskMark Utility , there are many tools available in the market depending on the type of Guest-OS .\nI picked up one of a windows server-2k16 VM , presented two new hard-disks of size 2GB each , applied the vSAN-Default-Storage-Policy on the first vmdk which is nothing but FTT=1 ,SW=1 and FTT=1 ,SW=2 storage policy is applied on the second disk. Next I ran similar workload test on both these drives , it is clearly evident that the writes are always benefited when the components are striped across multiple capacity tier SSD drives .\nRunning Crystal Diskmark against two drives! ![](/images/how-storage-policy-affects-performance-on-vsan-crystal_diskmark-exe-1.png) - Download and extract “CrystalDiskMark6_0_2.zip” , run the 64bit file ![](/images/how-storage-policy-affects-performance-on-vsan-Result_run_ftt-1SW1-1024x579.png) - Configure Test Settings (queue and threads)and run the benchmark tool against the first drive with** SPBM FTT=1 and SW=1 ** ![](/images/how-storage-policy-affects-performance-on-vsan-Result_run_ftt-1SW2_final.png) - Run the same benchmark test against the second drive which use **SPBM FTT=1 and SW=2 ** From the above tests we see that the J:Drive which is with higher stripe width defined on the storage policy yields higher throughput especially in terms of writes which might be critical for most of the applications which are I/O intensive , also read is benefited since the reads are also served by two different capacity tier drive directly to the VMs (Only in All-Flash).\nIt is always recommended to run HCIBench and Guest level bench mark tests with different tool which can generate a very close/similar workload to that of real life application and find the sweet-spot (right storage policy) which yields best performance for the specific use case.\nNOTE: A storage policy which is best suited for Application-A will not be the best storage policy for Application-B , its always recommended to run different tests of each application to find the correct storage policy . At times there will be contention at the virtual disk level because of the type of virtual storage controller like LSI-LOGIC , you might want to use PV-SCSI controller to get higher queue depth upto 256 for heavy VMS , see KB : https://kb.vmware.com/s/article/2053145 .\nIf you enjoyed this post, I’d be very grateful if you’d help it spread by emailing it to a friend, or sharing it on Twitter or Facebook. Thank you!\n","link":"https://blogs.hareeshkg.bid/post/how-storage-policy-affects-performance-on-vsan/","section":"post","tags":[],"title":"How Storage Policy affects performance on vSAN?"},{"body":"","link":"https://blogs.hareeshkg.bid/archives/","section":"","tags":null,"title":""},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-troubleshooting/","section":"categories","tags":null,"title":"Vsan Troubleshooting"},{"body":"We all know that there is a newer version of HCIBench 2.0 which was announced last month I was able to deploy and run this on one of the All-Flash Clusters , all new changes significantly seemed to have improved user experiences from the previous version . I request you to go thru “Detailed Guide for HCIBench-1.6x ” if you are looking for steps to deploy and configure HCIBench . Also better understanding the use case for HCIBench see : How to Achieve Best Performance Using HCIBench .\nHere are some of the highlight changes seen with latest version HCIBench 2.0\nAll-New Clarity-UI for this newer version of HCIBench-2.0 in parity with all VMware Product UIs which looks very niche\nAdded a new native FIO based workload generator in addition to the existing VDBench workload Generator\nOption for choose between two BenchMarking Tool within HCIBench-2.0\nOne of the Best additions to this newer versions of the HCIBench is live performance monitoring while the HCIBench is being run using open source Grafana and we still get the offline Observer graphs for analysis after the test finishes .\n- ![](/images/whats-new-with-hcibench-20-hcibench-2.0_deploy.png) Deploy HCIBench same as previous\nEdit and save Appliance Config\nStart HCIBenchmark test\nWait for tests to begin after deployment finishes\nClick HyperLink to view Live Graphs\nLive Grafana graph which refresh every 5 seconds\nReview final result and Observer data similar to previous releases\n","link":"https://blogs.hareeshkg.bid/post/whats-new-with-hcibench-20/","section":"post","tags":[],"title":"Whats new with HCIBench 2.0?"},{"body":"This guide will walk you thru all the new features about SRM 8.2 and vSphere Replication and steps to install , configure , manage and migrate to SRM 8.2 . Also covers all the new functionalities around vSphere Replication Aplliance 8.2\nHow to Install , Configure and Manage SRM 8.2 and VR 8.2 ? How to migrate from Windows based SRM to SRM-Appliance? Methods to install SRA on the new SRM-VA. How to configure replication for Encrypted Virtual machines? I am really exited about the new version of Site recovery manager 8.2 which now comes with an Appliance version and bring in “All New Experience” to the administrators to easily deploy and manage SRM .\nWhats New with SRM 8.2 and VR 8.2?\nI highly encourage you to go thru link for all additional details around SRM/VR 8.2 . For official release notes see Release-Notes-SRM-8.2\nSRM is now available as appliance an d uses the VMware standard Linux based Photon OS which always required to be deployed on top of a windows box as an exe file. This model of deployed reduces a lot of time to deploy and configure as we dont need to install an windows machine in this case and the database is always embedded. SRA adapters from the storage vendors are all uploaded as container images and they run as a docker container within the SRM-Appliance . Vsphere Replication 8.2 is now capable of replicating encrypted virtual machines and now can take advantage of storage policy based management , which deeply integrates with products like vSAN . Easy and simple way to migrate from windows based SRM to Appliance by exporting all the configuration using export utility. Large number of enhancements added to the Public API , which helps products like vRO to automate things better. SRM pack for vROPS has some enhancements for advanced reporting . New VR pack announced for vROPS . Great enhancements made for the SRM/VR UI with dark mode theme compatibility. SRM and VR 8.2 now supports NSX-T Improved Security and Flexibility with Support for Encrypted VMs Enhanced Security with vSphere Replication traffic encryption SRM Content Pack for vRealize Log Insight Download, Install and Configure SRM-Appliance Download SRM8.2 and Download VR-Appliance8.2 and follow the below guided screenshots for deploying SRM appliance .\nSince vSphere Replication Appliance deployment has always been an appliance and there is no difference in the deployment process , I have skipped the steps to deploy VR appliance and covered SRM-Appliance deployment here .\nSteps are as follows :\nStart with the OVF deployment similar to VR appliance and fill in OVF configuration details Power on and Complete Configuration Similar to VR appliance we will be logging on to the SRM-Appliance VAMI page i.e https://SRM-IP:5480 and login with the admin/password which is slightly different on the VR appliance where we used to use root/password . Once you login perform the configuration similar to VR where we provide PSC/VC authentication to SRM and perform plugin installation and solution user registration .\nAppliance additional Configuration Views\nWe may choose to deploy two new appliances on either sites and configure fresh setup of SRM-Appliances , however if you have an existing SRM-8.1 setup running on a production environment and would like to move to the appliance version of SRM 8.2 , you will be using the Migration utility and export the data into the SRM 8.2 -virtual appliance\nSite Pairing Site Pairing procedure remains similar to the previous version . Please make sure that plugins are visible and configured on both side before you move forward with site pairing , I have added screenshots for reference .\n*Migrating from SRM -Windows to SRM-Appliance * To migrate any SRM-windows version to be able to migrate to SRM-8.2 appliance , we have to first upgrade the windows SRM version to 8.2 , and then initiate the Migration Process .\n![](/images/how-to-installconfigure-and-migrate-to-site-recovery-manager-82-Old_SRM_StopProcess-1024x641.png) **Step1 :** Login to the current SRM-Windows VM running SRM-8.1 and stop the SRM service . ![](/images/how-to-installconfigure-and-migrate-to-site-recovery-manager-82-Migrate_1_Export_data-1024x554.png) **Step2:** Launch command prompt change directory to : C:\\Program Files\\VMware vCenter Site Recovery Manager\\bin and ruin command : .\\export-srm-data.bat . Copy the SRM-data folder to an accessible location and shutdown the old SRM-windows box after making a note of its hostname and IP address . ![](/images/how-to-installconfigure-and-migrate-to-site-recovery-manager-82-Migrate_2_import_data.png) **Step3 :** Deploy a a new SRM-Appliance with the same IP/FQDN and hostname , complete the deployment , however dont proceed with the VAMI page configuration . Copy the download SRM-data folder into the appliance using winscp to “**/home/admin/**“and complete the import process , reboot the appliance . Note:* The Restore procedure needs to be run as super-user “root” , hence login to appliance via SSH using user admin , later switch user to root to follow further Import procedure . Also import Procedure logs can be viewed at path : /var/log/vmware/srm/import-srm-data.log\nPost Import Procedure Now that the the DB and configurations are restored on the new SRM-VA it will still need a reconfigure-pairing between the sites , the reason being that the new SRM-VA will use the default 443 port to expose SRM server/service to the vcenter unlike the previous windows based SRM i.e SRM lookup service end point will change from “https:// SRM-Windows-IP:9086/vcdr/vmomi/sdk” to “https:// SRM-Appliance-IP:9086/drserver/vcdr/vmomi/sdk” , hence post migration complete the Re-Pair operation from either of two site’s SRM-UI (https://SRM-IP/dr) .\nUnity-VSA (forABR) You might already read thru my other blog where I have demonstrated steps to deploy Unity VSA for array based replication with SRM . I will be using the latest version of Unity-VSA from Dell-EMC on this illustration process. Steps to download and configure unity vsa remains the same as that of previous versions however download URL and license generator URL have changed . Please find the deployment and install process here : How to configure Pool, Luns, replication and SRA for SRM -UnityVSA . New Download URL : https://www.dellemc.com/en-us/products-solutions/trial-software-download/eula/unityvsa.htm New License-Gen URL : https://www.dellemc.com/hi-in/auth/elmeval.htm\nMake Sure hosts are connected to respective Unity-VSA\nConfigure Replication connection between two VSA\nConfigure Replication for one of the from PR to DR\nFollow thru Wiz*\nFollow thru Wiz*\nComplete Replication config\nVerify Replication Session\nCheck Lun Status post Replication Config\nSRA Installation Procedures SRA can be installed in two methods either from the default VAMI page where we have a provision to upload the SRA-tar.gz file or can be installed via CLI .\nSRA-UI-Installation and Array-Pair configuration are illustrated below . SRA-CLI installation and cleanup procedure is also very simple , download the SRA from the respective array vendor webpage , copy the file into the SRM-VA using winscp into folder /home/admin , run command : “docker load -i \u0026lt;sra-file.tar.gz\u0026gt;” .To uninstall , you will need to make sure that there are no containers running on top of the docker image from SRA vendor , make sure to stop and delete all the docker containers and finally delete the docker image nothing but SRA .\nStorage Policy Mapping for VMs As you have read in the new features list we have now enabled vm storage policy mapping between the sites , this comes very handy when you have two sites running vSAN and you want to make sure that the storage policy at the source and target are always same . This storage policy really matters when it comes to performance comparison between the two sites i.e A VM running on the production site with RAID-1 Storage policy with higher Stripe width will need the same storage policy and stripe width requirement on the DR site and it should not be using other storage policy which can compromise the performance of your VM . Hence with SRM 8.2 along with your regular object mapping such as network,folder,resource,placeholders we now have Storage Policy mappings which ensures storage policy compliance for encrypted and vSAN virtual machines . I have added screenshots for configuring replication and protecting the VM using SRM below .\nReplication of Encrypted Virtual machines\nVM encryption allows the per-VM encryption capabilities, thereby making VM data on vmdk disks more secure. SRM and VR 8.2 now support protecting, replicating, and recovering VMs that are protected by VM encryption , however older version only supported protecting VMs using vSAN encryption .\nReplication traffic between the hosts and the VR appliance can be also encrypted on per VM basis .\nConfiguring Replication on Encrypted VMs\nIn order to protect and configure replication for VMs which are encrypted between two sites , we need to make sure that the same KMS server is configured for both sites also with the same “KMS-Cluster Name” if the KMS-Cluster names are different we will see an error while configuring replication . Also we encrypt VMs which already replicating to another side , to configure encryption and replication on such VMs , you will need to stop replication and discard target VMDKs for that VM , configure encryption for that VM via SPBM and re-replicate the VM to target . Also to note encrypted VM cannot pick a non-encrypted seed at target .\n","link":"https://blogs.hareeshkg.bid/post/how-to-installconfigure-and-migrate-to-site-recovery-manager-82/","section":"post","tags":[],"title":"How to Install,Configure and Migrate to Site Recovery Manager 8.2 ?"},{"body":"Here is a detailed guide on how we may approach ways to optimize a vSAN cluster to deliver best performance on a cluster and find your answers to your questions.\nHow do we assess vSAN performance ?\nWhat can I expect from a specific hardware?\nHow best can we tweak vSAN clusters to achieve best performance ?\nWhat are the tools which we can use to use to benchmark vSAN performance ?\nI highly encourage you to go thru the guide for Deployment Guide for HCIBench before you read this blog about running HCIBench Tests .\nI would like to write this guide in three sections .\nPre-requisite for the vSAN cluster before performing an sorts of benchmark tests and methodology\nHow to use HCIBench test with different parameter files to find the sweet-spot.\nResult comparison between different tests and choosing an optimal SPBM and workload set .\nPre-Requisites All-Green vSAN Configuration Health Checks Please make sure that we are not seeing any sorts of health check failures under Cluster Monitor vSAN Health , this will avoid any sort of deviation due to H/W compatibility , driver\u0026amp; firmware for controllers/disks , unsupported controllers etc .\nNever run Performance bench marking application when the cluster is already in production , This means that if there were running production vms in the cluster and if you start any benchmarking tests , this will lead to performance issues with the production VMs . Also the test results will not be accurate since cluster was already loaded with production VMs . If you would like to still run HCIBench on a production cluster , its recommended to power down all the VMs over maintenance window and then run HCIBench to get near accurate results . Note* : Make sure write-buffer free % is greater than 90% before you run a test .\nMethodology To achieve a satisfactory vSAN performance results , one must be aware of what kind of applications this cluster is expected to run , the type of drives used and have a reasonable expectations which will result in a satisfactory result .\nPlease understand that the results what we see with any benchmarking tool (Synthetic workload) may not be the same results when we run the real life workload , the results may vary . These parameters and workloads defined on the tools we use may be completely different with your real life application . Hence its recommended to use the right tool , though HCIBench tool is the good way to start to get initial benchmark numbers on the cluster .\nDownload and Run vSAN HCIBench Please read thru the complete guide to download and deploy HCIBench here where I have discussed the reason why we should choose use HCIBench to run performance benchmarking on vSAN datastore and how do we download and deploy .\nOnce the HCIBench is deployed we will need to run different test by changing parameters, perform tweaks to achieve best performance . We will see what are the test we can run within HCIBench to find the perfect sweet-spot to get best performance on a cluster.\nRecommended Workflow for HCIBench The workflow which I usually follow is to run the easy-run on the HCIBench first , get the results use that as a reference number and then start playing around with the parameters , storage policy and run different test using the same VMs which was previous deployed by the easyrun bench mark test to reduce waste of time redploying the VMs all over again , compare the results change up the parameters again until we can find a satisfactory number or the sweet spot which gives us best numbers .\nRunning HCIBench on a 4Node-All-Flash Cluster I just got my hands on a 4-Node-All-Flash supermicro cluster which I will be using the HCIBench to find the sweetspot to get the performance bench result .\nHardware Configuration ** Recommended to copy below content to a Notepad for aligned text**\nserver model and BIOS version Supermicro SYS-2028TP-HC0TR | BIOS: 2.0 WebHCL compatibility check vcgLink = 'https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=server\u0026amp;productid=41008' picture = 'https://www.google.com/search?q=Supermicro%20Computer%20Inc%20SYS-2028TP-HC0TR\u0026amp;source=lnms\u0026amp;tbm=isch\u0026amp;sa=X' - CPU type Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz # Link: 'http://www.google.com/#q=E5-2650+ark.intel.com' Number of packages (pCPU):2 Number of cores (total):20 Hyperthreading state: 3 -\u0026gt; enabled\n-** CPU microcode revision** :\n1 Current Revision:0x0000003d 2 3\u0026lt;span style=\u0026#34;text-decoration: underline;\u0026#34;\u0026gt;**- Network Interface Card** \u0026lt;/span\u0026gt; 4 2X 10-Gigabit SFI/SFP+ Network Connection (Used for vSAN) 5 6\u0026lt;span style=\u0026#34;text-decoration: underline;\u0026#34;\u0026gt;**- Host Bus Adapter**\u0026lt;/span\u0026gt; 7 lsi_msgpt3 Avago (LSI Logic) Fusion-MPT 12GSAS SAS3008 PCI-Express 8 9**SSD-Drive details from SSH :** 10 11[root@esxi03:~] esxcli storage core device list |grep -i \u0026#39;Devfs Path\\|vendor\\|Model\\|Revision\\|Raid\\|size\u0026#39; | egrep -v Queue |sed \u0026#39;N;N;N;N;N;s/\\n//g\u0026#39; 12 Size: 763097 Devfs Path: /vmfs/devices/disks/naa.55cd2e404c211107 **Model: INTEL SSDSC2BB80** Revision: 0140 13 Size: 763097 Devfs Path: /vmfs/devices/disks/naa.55cd2e404c211073 **Model: INTEL SSDSC2BB80** Revision: 0140 14 Size: 763097 Devfs Path: /vmfs/devices/disks/naa.55cd2e404c212d55 Vendor: ATA ** Model: INTEL SSDSC2BB80** 15 Size: 190782 Devfs Path: /vmfs/devices/disks/naa.55cd2e404c20d56d Vendor: ATA **Model: INTEL SSDSC2BX20** 16 Size: 763097 Devfs Path: /vmfs/devices/disks/naa.55cd2e404c2114c4 Vendor: ATA **Model: INTEL SSDSC2BB80** 17 Size: 763097 Devfs Path: /vmfs/devices/disks/naa.55cd2e404c212a02 Vendor: ATA ** Model: INTEL SSDSC2BB80** 18 19Cache Tier VCG Link :[https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd\u0026amp;productid=40328](https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd\u0026amp;productid=40328) - Performance Class:\tClass C: 10,000-20,000 writes per second 20Capacity Tier VCG Link : [https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd\u0026amp;productid=40540](https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd\u0026amp;productid=40540) - Performance Class:\tClass C: 10,000-20,000 writes per second 21 22DG layout : 23 24[root@esxi03:~] vdq -i 25[ 26 { 27 \u0026#34;SSD\u0026#34; : \u0026#34;naa.55cd2e404c20d56d\u0026#34;, 28 \u0026#34;MD\u0026#34; : [ 29 \u0026#34;naa.55cd2e404c211107\u0026#34;, 30 \u0026#34;naa.55cd2e404c211073\u0026#34;, 31 \u0026#34;naa.55cd2e404c212d55\u0026#34;, 32 \u0026#34;naa.55cd2e404c2114c4\u0026#34;, 33 \u0026#34;naa.55cd2e404c212a02\u0026#34;, 34 ] 35 }, 36] Easy-Run Test and Results : Setup the configuration page , perform a validation test . Here in this test I have used the vSAN default storage policy and the cluster has dedup and compression enabled .\n[ ![](/images/how-to-use-hcibench-and-achieve-best-performance-on-vsan-Config-setup-1-150x150.png) ](/images/how-to-use-hcibench-and-achieve-best-performance-on-vsan-Config-setup-1.png) [ ![](/images/how-to-use-hcibench-and-achieve-best-performance-on-vsan-Config-setup-2-150x150.png) ](/images/how-to-use-hcibench-and-achieve-best-performance-on-vsan-Config-setup-2.png) [ ![](/images/how-to-use-hcibench-and-achieve-best-performance-on-vsan-Config-setup-3-150x150.png) ](/images/how-to-use-hcibench-and-achieve-best-performance-on-vsan-Config-setup-3.png) Run Test and Review results I was able to run different tests with this cluster and found that the RAID-5 Performance was pretty decent when compared to RAID-1 considering the fact that the data components was placed across all four hosts when compared to the fact that RAID-1 will mostly pick two hosts to place the data components . You may Download the Results for offline view .\nBased on the test results : we see that the test which was run with 8 vmdks with 4gb capacity tier drive with 70% read , 100% random packets, 100% working set is a common test which was run with various storage policies . Below is what we understand from the test results .\nRaid-5 , FTT-1,SW-1- 8vmdk-100ws-4k-70rdpct-100randompct-4threads gave us the best performance among all the test which was run in this environment , in fact I had this run two times to see if the results are consistent .\nRaid-1 , FTT-1,SW-4 – 8vmdk-100ws-4k-70rdpct-100randompct-4threads ranks seconds for the best performance , however I expected this result to be better than the Raid-5 Sw-1 test . May be changing the size of the vmdk would give us different test results .\nRaid-5 , FTT-1,SW-4 , comes next with the same working set , however I expected this test to give more IOPS/Throughput when compared to Raid-5 , FTT-1,SW-1\nWhat do we understand from graphs ? Here I just picked up one of the test results to explain what we can understand for the test Raid-1 , FTT-1,SW-4 – 8vmdk-100ws-4k-70rdpct-100randompct-4threads-stats.html file which display graphical information which was collected by observer data within the HCIBench VM .\nvSAN Client – This graphs displays the throughput and latency values seen at client side which is from Virtual machine perspective . we see that the latency value is consistent and less than 4ms across all the hosts . We see that the IOPS,Throughput(Bandwidth) and latency is slightly higher on host-10.109.10.104\nDOM-Owner – This graphs displays the throughput and latency values seen at layer just before IO traverses thru vSAN back-end file-system which is the object-owner level latency . The above graphs values are almost similar to what we see at the vSAN-client level .\nvSAN-Disks – This graphs displays the throughput and latency values seen at the beginning of vSAN-backend layer called DOM-Component Manager From the above graphs we see that the number of IOPS and Throughput dropped over the period of the during the test and the outstanding number of IOPS were also spiking up on the host – 10.109.10.104 . This will need further investigation .\nvSAN-Disks(DeepDive)- Shows the graphs from individual hosts against the devices which participating in the vSAN cluster on respective hosts . From the above graphs and comparison between the results from all four hosts . The physical Drive latency values are all less than 200 microseconds . we see that for the host-04 was seeing IOPS greater than 60k while other hosts had less than 40K . We also see that the write buffer consumption on the Cache-SSD drive was close to 70% towards the end of the test while others stayed within 50% . We also see that the SSD started to elevate the data from Cache to Capacity drive soon after we cross 30% write buffer consumption .\nFrom section vSAN-More and congestion , we see that the Log congestion against host-04 was very high , this log congestion might be due to the fact that the number of IOPS against host-04 was very high and the write logs from above graphs was constantly increasing , hence the host had to build up log congestion to get some time to elevate the data from write buffer down to capacity tier drive . Conclusion : From all the above graphs we can conclude that the IOPS/Throughput which started to drop at the vSAN-Client layer on host-04 was because of the fact that the SSD drive started to build Write-Log congestion as high as 180 , due to fast write buffer fill up . It is possible that the VD-bench VMs which were running on this host generated very high IOPS when compared to other hosts , or the write buffer was not cleared before the test was started .\n*Note: **The test result which we saw with various tests thru HCIBench are with synthetic test workload , the parameters may vary with real life applications / workload . It is recommended you run different test from within the guest VM which can generate workload similar to that of production application .\nI can discuss a few guest level benchmarking tools in my next Blog . I hope you found this useful .\n","link":"https://blogs.hareeshkg.bid/post/how-to-use-hcibench-and-achieve-best-performance-on-vsan/","section":"post","tags":[],"title":"How to use HCIBench and achieve Best performance on vSAN ?"},{"body":"Here is a quick and easy guide to convert a normal vSAN cluster to a stretched cluster \u0026amp; converting a stretched cluster to a regular vSAN cluster .\nConverting a Standard vSAN cluster to Stretched Cluster I can categorize the tasks we will need to perform when we convert a normal cluster to a stretched cluster into three sections and I would also quote some of the examples where this will be your requirement in real life situation .\nPre-requisite\nSteps for converting to stretched cluster\nPost conversion checks\nSome of the Use Cases : Consider that a business has grown and you are trying to stand up another co-location which is within 5 ms LTT between the sites , see LINK however you don’t want use this site as a Disaster-Recovery/ Fail-over site rather use this in an active-active configuration where users from both offices can run workload simultaneously on both sites . In such cases converting a regular vSAN cluster to a stretched cluster will be very useful .\nIn some cases where we have a physical limitation of server racks , lets say we have two server racks which hosts 4 Nodes per rack and we would like to withstand a rack failure , however we cant configure just a regular fault domain as we dont have a third rack to configure a regular cluster with fault domain . In such cases we can have a witness hosted on a different host/cluster/cloud have the two server rack put in two separate fault-domain to tolerate rack failures .\nPre-requisite : A backup of all running VM is a must as there is significant risk involved for the objects which back your virtual machines during this transition.\nValidate the cluster for all best practices and requirements for stretched cluster , do read : LINK-1 , LINK-2\nYou have a different host/non-vSAN stretched cluster /cloud provider to host your witness appliance or witness host .\nYou do not have any vSAN iSCSI targets configured on your current vSAN cluster , since vSAN iSCSI targets are not supported with vSAN-stretched cluster you will need to disable this , having vSAN iSCSI target enabled on stretched cluster can cause PSOD on the hosts part of the cluster .\nYou will be loosing a significant amount of datastore space on the vSAN Datastore during this transition , make sure your cluster is not nearing its full capacity ensure 30% or high free space availability .\nTo be able to remove and add the hosts to a different fault domain , you may have to re-apply policy on certain VMs whose storage policy might not comply when we are about to remove the hosts . It is recommended to change storage policy of few bunch of VMs at a time as there will resync operation involved here , changing policy on multiple VMs at a time might fill up the datastore since certain storage policy change will require double the amount of space consumed during the transition .\nSteps for converting standard vSAN cluster to stretched cluster You may be needing to physically disconnect one or more hosts from the cluster in-order to perform a re-location of the hosts physically to your other site .\nHere in this example I have got 4 node vSAN cluster running vSAN 6.7 with vCenter server running 6.7 U1 , the steps used should be the similar for all version of vSAN cluster which supports stretched cluster .\nIn my setup (These are nested ESXi hosts not physical ones) I would like to use esxi-1-1 and esxi-1-2 on primary/Preferred Site and esxi-3, esxi-4 on the Secondary site . I have added a witness appliance esxi-5 to the Datacenter as a host which is deployed on a different cluster.\nFrom the above screenshot we see that only dedup and compression was enabled on this cluster and vSAN iSCSI target service is not enabled . Step1: Before you start taking any actions , you will definitely need to make sure that all the VM storage policy are compliant and vSAN health reports no errors under “Data” section . Lets look at the two VMs here and their placements to get an idea how the components are placed before we place the two hosts in maintenanceMode .\nFirst VM : Linux-2 , has storage policy set to Ftt-1 (RAID-1) , where the components for VMDK and VM home folder are on hosts 1 , 3 and Host 4 .\nSecond VM : VM-FTT-0 , has storage policy set to Ftt=0 (Raid-0) where the components for VMDK and VM home folder are on Host-03\nStep2 : Place the host esxi-3 and esxi-4 in maintenance mode with ensure accessibility , which means that the VMs which are on this cluster will move to the hosts : esxi-1 and esxi-2 . Also if there were any objects which would go inaccessible without host esxi-3 and esxi-4 will be rebuilt onto hosts esxi-1 and esxi-2 (like Ftt-0 objects) before the hosts completely enter maintenance mode and later you may disconnect the hosts move them physically to the co location .\nPlaced host-03 and host-04 in maintenance mode and look at the component placement . From the above screenshots its clear that VMs which have FTT-1 will go non compliant as they dont have any more hosts to place the components and FTT-0 objects from the hosts which were put in maintenance mode have moved to the hosts which are still accessible . Note: As explained in the pre-requisite if you have plans to keep 3 or more nodes on each side , you will retain certain amount of hosts and place the rest of the hosts in maintenance mode , however with vSAN if we have 3 or more hosts accessible , vSAN will try to rebuild absent components on to the hosts which are not in MM mode . This may be a good thing or can cause disaster in certain cases .\n**If we have enough free space on the vSANdatastore after placing half of the cluster hosts in MM mode we should allow vSAN to rebuild absent components so that you can protection against failure of disks/diskgroups or even hosts . **\nIt will be a disaster when a huge resync triggers for the absent objects and we dont have enough space on the datastore to accommodate the resyncing objects , there will be no way to stop the resync once it triggers , we will need to add the hosts which were removed immediately back into cluster to avoid any sorts of downtime, allow resync to finish place the hosts which needs to move out back in maintenance mode with ensure accessibility and adjust ****the repair delay to a value which is larger than the time it would take for you to move the hosts to the other site and restart the clomd service for the value to immediately take effect .\nHere in my case I only have two hosts which are still contributing to vSAN storage , hence vSAN wont start to rebuild any absent components since I dont have enough hosts to comply to the policy.\nIf we have more than 3 hosts still active and it will take me more than 2 days to bring other two hosts back in network , We will be needing to perform the following task to stop any resync triggers which can potentially cause a datastore fill-up . Approximate time value for clomd which I need to set considering an extra day will be 32460 = 4320 minutes . So I will ssh to all the hosts and run the commands : esxcfg-advcfg -s 4320 /VSAN/ClomRepairDelay and /etc/init.d/clomd restart\nAssuming I have more than 3 nodes and I can still go complaint on the storage policy on all my VMs while the rest of the hosts are being moved to secondary site . I will force full re-protect all of my VMs soon after I place the required host in maintenance mode by clicking on repair objects immediately button under Cluster⇒ Monitor ⇒ vSAN⇒Health ⇒ Data ⇒ vSAN Object health , this will help me tolerate any additional failures within the cluster while we stand up rest of the hosts on the other site . **Step3 : **The following tasks needs to be followed to complete stretched cluster conversion :\nReconnect the hosts back to cluster , exit maintenance mode Configure fault domain and stretched cluster. Exit maintenance mode on the hosts which were added to secondary fault domain\nRepair objects immediately , review placement for confirmation If there were any non-critical VMs which were set to ftt-0 to address space crunch before configuring stretched cluster those VMs can be now re-protected with FTT-1 policy as we have regained available space on the vSAN datastore since the hosts are now contributing space to storage .\nConverting a Stretched vSAN Cluster to a regular vSAN cluster There may be situations where we need better space efficiency on vSAN datastore and stretched cluster is not working well , there may situations where we need vSAN iSCSI targets for windows failover clustering and sometimes will find the need of a separate host the will be unnecessary . In these situations a Stretched vSAN cluster can be covered to a regular vSAN cluster .\nPre-requisite : A backup of all running VM is a must as there is significant risk involved for the objects which back your virtual machines during this transition.\nMake sure all the VMs are compliant with their storage policy , make sure there is no ongoing resyncing objects , all vSAN health checks are clean.\nRead official page here\nSteps for converting Stretched vSAN cluster to standard cluster The following tasks needs to be followed to complete conversion of stretched cluster to standard vSAN cluster:\nDisable stretched cluster Cluster⇒Configure⇒vSAN⇒Fault Domains⇒Stretched Cluster⇒Disable. Destroy the fault domains. Repair all the objects immediately. Change the storage policy for few VMs at a time if required to achieve any additional space efficiency.** This step is optional\nRemove the witness node from inventory to avoid any confusion.** This step is optional\nThank you for taking time to understand process of converting standard cluster to stretched cluster and viceversa . Feel free to comment below if there were any questions .\n","link":"https://blogs.hareeshkg.bid/post/how-to-convert-standard-vsan-cluster-to-stretched-cluster-viceversa/","section":"post","tags":[],"title":"How to Convert standard vSAN Cluster to Stretched Cluster \u0026 viceversa"},{"body":"I have been asked about re-configuring replication for a VM without deleting the destination folder for a VM which is already replicated . In a traditional storage , we were simply renaming the folders on the destination datastore (Can be a local or a remote site) , we used to re-name the folder at the target site , perform a force stop replication at source and target site and simply re-configure replication to the same folder using existing seeds to avoid a full-sync between the two sites .\nThings can get very complicated with vSAN as we donot have an option to directly rename the folder ,the folders which we create is merely a symlink to a UUID (NameSpace folder) on the vSAN datastore and hence we cannot rename the UUID .\nLets see an example where a VM is being replicated from primary site to the DR site using VR 8.1 .** Following Example and steps applies with vSAN 6.6.x / vSAN 6.0.x with vSphere Replication 8.1.x and vSphere Replication 6.5.x .**\nSetup Three node vSAN cluster running vSAN 6.7 at both Primary and Secondary Site . We are using VR version 8.1 in this illustration . We have a VM called win7 which is being replicated from source to Target Site .\nVM we are going to work with is win7\nLets look at the replication configuration from Source and the destination objects at the DR site . Here we see the VMDK which is being replicated from source host p-h1.vsensei.local for the VM “win-7” which has a vDisk called win7.vmdk , replication ID for this VMDK is RDID-16d6932c-0b80-4838-bb36-f52fabf8fd97 (see on the left hand side under get-config details using command “vim-cmd hbrsvc/vmreplica.getConfig VM-ID”). On the Right side we see the datastore⇒NameSpace path where the VM is being replicated to . This contains the hbrdisks (vmdk) with respective RDID disk/ (Journal disk) i.e “hbrdisk.RDID-16d6932c-0b80-4838-bb36-f52fabf8fd97.18.39977055102374.vmdk” and the respective base disk for that VM which is in this case is again win7.vmdk (At the Target Site)\nJust to show you the object attributes for the NameSpace which correlates to the vm-folder with command “esxcli vsan debug object list” against the UUID for the actual vmdk object where this pointer file “win7.vmdk” is pointing to at the DR site .Please also note that in the vSAN datastore , the pointer file will not be pointing to a “VM_NAME-flat.vmdk” like on a VMFS file system where the data gets written it actually points to an object on the vSAN file system as seen below . hence the pointer file will usually be in bytes , same applies to the HBR-RDID files who also point to a vSAN object in the backend .\n[root@s-h1:/vmfs/volumes/vsan:52f056aba5df4658-8e963893c4dd02a8/1b80b15b-ef89-d8ac-63c8-005056b81125] cat win7.vmdk | grep -i vsan RW 20971520 VMFS \u0026quot;vsan://1e80b15b-4a58-81f4-db87-005056b880b5\u0026quot;\nObject UUID: 1e80b15b-4a58-81f4-db87-005056b880b5 Version: 6 Health: healthy Owner: s-h3.vsensei.local Policy: hostFailuresToTolerate: 1 CSN: 1\nConfiguration:\nRAID_1 Component: 1e80b15b-2cbe-a7f5-a942-005056b880b5 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 528b4edb-a6a9-258a-572d-e75eba525740, Disk Name: naa.6000c29e66232078e9eca8c25e5b0494:2 Votes: 1, Capacity Used(B): 10007609344 (9.32GB), Physical Capacity Used(B): 9906946048 (9.23GB), Host Name: s-h1.sensei.local Component: 1e80b15b-86b6-a9f5-f986-005056b880b5 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 5247c151-75bb-ba7d-cd70-620452e0751c, Disk Name: naa.6000c29ee707b1c208ab912d7f826646:2 Votes: 1, Capacity Used(B): 10007609344 (9.32GB), Physical Capacity Used(B): 9906946048 (9.23GB), Host Name: s-h3.vsensei.local Witness: 1e80b15b-51cc-aaf5-2984-005056b880b5 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 525dacfc-8968-4af9-2961-fbc18247c3fb, Disk Name: naa.6000c294eba0d6d3c3be8c3b60bda715:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: s-h2.vsensei.local\nType: vdisk Path: /vmfs/volumes/vsan:52f056aba5df4658-8e963893c4dd02a8/1b80b15b-ef89-d8ac-63c8-005056b81125/win7.vmdk (Exists) Group UUID: 1b80b15b-ef89-d8ac-63c8-005056b81125 ⇒ is the NAMESPACE Folder or the VM Folder UUID . Directory Name: (null)\nNow that we know how the replication is working between the two sites where the vsphere replication happens between the two sites on a vSAN datastore at both primary and DR site . We shall see how we can re-configure replication without renaming the folder as this is not an option on vSAN datastore due to the nature of the filesystem .\nStep 1 Make a note of the namespace folders (VM working directory where the vmx , vmdk etc are located ) at both primary site and the DR site .\nStep 2 Force stop replication at the protected site or the Primary site .\nStep 3 Create a dummy/temporary folder on the vSAN datastore at the Target or the Destination site and copy all the content from the current target replication folder to the dummy folder .\nNote : It is expected that the replication status for the VM at the target-site in the incoming direction will show up as “Error state”\nI preferred the SSH method to copy the content of the files from the current target destination folder to the Dummy Folder .\nStep 4 Once I have copied all the folder content from the target folder to the dummy folder , I delete the content inside the original target folder , in my case it is the files inside folder “win7” at my target site .\nStep 5 Now that we have moved all the files from the original folder , we are safe to force stop the replication from the incoming direction at the target site , since there are no files / vmdks on the original folder there will be no deletion task against the vmdk objects , vcenter server will also report a vmdk deletion task failure .\nStep 6 Now that we have successfully stopped replication from both source and the target site . We are good to copy just the actual vmdk object , here in this case win7.vmdk from the “Dummy” folder back to original target folder . The reason we dont have to copy the other files is because when we reconfigure replication from the source site , we will create a new RDID and respective HBR-RDID vmdk for the replication session , the current files are no longer valid/required .\nFrom the above screenshot we can see that the win7.vmdk is copied back to to the win-7 folder (Original target Folder) .\nStep 7 We can now safely re-replicate the VM from the source to the same target folder using existing seeds , avoid a full-sync replication and only replicate the delta-changes to the Target .\nIt is good to check the status of the replication from the command line as we will still see the status on the GUI as full-sync however vSphere-Replication appliance will only be comparing the checksum data between the two VMDKs at the source and target once it finishes the checksum comparison , the replication status will go back to read state . You will also see that the amount of data transferred to the Target site will be 0 bytes.\nRecap We can safely move the files from original target folder to a dummy folder and these file copy/movement should finish in couple of seconds as there is no data to actually copy , all files are just pointer files and not actual vmdk.\nWe can safely delete the content on the original target folder\nForce stop replication from both Source and Target site , we will see the vmdk deletion task fail at the vcenter , because the Original target folder is empty .\nNext copy/move just the base disk pointer from the target folder back to the Original target folder .\nRe-configure replication for the same VM using existing seed and vsphere replication appliance will only compare the checksum before it starts to seed again to the target vmdk .\nConclusion We are saving a ton of bandwidth usage between to sites as we do not have to do a full-sync to the target site , rather only do a simple checksum comparison and continue to re-replicate the VM using existing seed .\n","link":"https://blogs.hareeshkg.bid/post/how-to-re-config-replication-with-existing-seed-on-vsan/","section":"post","tags":[],"title":"How to Re-Config Replication with existing seed on vSAN"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vsan-general/","section":"categories","tags":null,"title":"VSAN-General"},{"body":"vSAN Support Insight was introduced last year with the release of vSAN 6.6.1 ( VC/ESXi 6.5 U1 Patch 02 and above) . It is a phone home system leveraging the existing vCenter Customer Experience Improvement Program (CEIP). Health, performance, and configuration information for VMware Support and engineering. Once enabled and verified, log information is pushed back to VMware’s Analytics Cloud on a regular cadence approximately every hour . No actual customer data is phoned home and user enter able or identifiable data (Hostnames, VM names, subnets, IP addresses, Mac addresses) are obfuscated . In this blog you will further see what data is seen by VMware support vs your Webclient View .\nHow does vSAN Support Insight help Customers? Resolve your support ticket much faster by avoiding wastage of time by collecting multiple logs and uploading it to the support ticket .\nHelp vSAN Engineering team analyze pattern of issues seen with different customers, helps them towards product hardening and development in future release .\nvSAN TSEs can view your environment with the data collected from VAC (VMware’s Analytics Cloud ) in obfuscated manner , provide recommendation/ action plan without a need of a remote session/webex to your actual environment and if you were entitled for a Premium Support Agreement , you receive Proactive Support from a TSE based on issues seen from VAC .For more FAQs , please visit : vSAN-SUPPORT-inisight-FAQ\nTo Enable CEIP program for vSAN Support insight , follow the steps available HERE What Telemetry Data is being sent to VMware? Security teams will be really concerned about the data which is being relayed to VMware via CEIP , well you call definitely see what data is being sent to VAC from your vcenter server . In my current test setup I have a VCSA .\nSSH into VCSA\nFrom the vSphere-Webclient (Flash based) , find out your cluster UUID , by going to Cluster ⇒ Configure ⇒ “Health and Performance” , under “Online Health Check” , you have “Support Tag” , where you will have two UUIDs sperated by colon “:” , example (918c59e1-7456-45e9-9575-9e18d074f480:523d5e56-05a4-d751-0c33-04ae7a42599b) , the first UUID is your Vcenter server UUID and the second UUID is your Cluster UUID .\nRun command: cd /var/log/vmware/vsan-health/ ,\nData collected by online health checks is written and gzipped to files ” cloud_health_check_data.json.gz” and ” vsan_perf_data.json.gz , seen below example for the UUID which we had made a note earlier for vcenter and cluster .\nAs we can see the JSON files are generated for vcenter as a whole and also for individual cluster within the vcenter server .\nYou can extract the json content by calling ” gunzip -k ” or view the contents by calling ” zcat “ Note that you need an “obfuscation map” (aka key) to make sense out of the data in terms of host-names / VM names / ip-addresses etc. Without that you can stare at the dataset all you want, but you won’t be able to relate it back to a customer. I would also add that we are not storing any VM/Workload data, it is configuration data / feature usage / performance data.All private data is obfuscated ,you can find the obfuscated data mappings in files \u0026lt;uuid\u0026gt;_obfuscationTableForHuman.json.gz\nWhat You See ? Lets use the same example for a vcenter server whos UUID is “918c59e1-7456-45e9-9575-9e18d074f480” and a Three node All-Flash which is my test environment who’s UUID is “523d5e56-05a4-d751-0c33-04ae7a42599b” .\nLets now assume that I am filing a support ticket with VMware for an issue/warning for this particular cluster . I have already signed up for VMware-CEIP and CEIP configured properly , this can be verified on the webeclient under vSAN health plugin .\nFrom the Above screenshot we can see that this is a a three node vSAN cluster and we have some warnings against vSAN Build Recommendation and some online health check . Now that I have filed a support request with the vSAN-Technical support , I will additionally share the vcenter UUID (Cluster ⇒ Configure ⇒ “Health and Performance” ⇒ “Online Health Check” ⇒ “Support Tag” ) along with my problem description .\nNow lets see what support team sees from their perspective . They simply enter the UUID of the vCenter server and Cluster UUID where the problem was reported and they will see a different view similar to vcenter , however no IP address or Virtual machine information , here are some screenshots for the same cluster and the data a TSE can view from their perspective .\nWhat Support See ? The **General **view contains details about the vcenter /ESXi versions and full specs of the server and vSAN configuration , it also contains limited vSAN cluster performance graphs and alarms seen on the cluster. It also has the list of storage profiles which were created for use with vSAN .\nConfig View shows all the features enabled on the cluster which includes vSAN-config , DRS , HA , Stretched cluster state , vSAN health and performance , number of fault domains , vSAN advanced parameter value sync between all hosts .\nThe vSAN-Health is exact set of data polled from the vSAN health plugin on the webclient , just that the data available on the VAC would be older than an hour from what we see on webclient . If we closely notice the health plugin is reporting the exact same warning what we saw earlier on the webclient .\nThe json export also contains the Disk group configuration and other information like the drive type,model and capacity information\nThe VM tab contains list of VMs and their head disk healthy summary , here as well we see all data is obfuscated .\nFinally the Performance Tab which is the most interesting feature , which uses the open source Grafana charts to show the perf data that was collected to VAC , which support can go back in time and investigate performance issues with help of different perf charts/parameters and custom time ranges.\nPlease note that you may share the Obfuscation map data which can be collected from health and performance tab to the support team for them to see the cluster, host and VM names as you see on the webclient .\nI highly recommend to watch the official VMware vSAN support insight Demo- here :\nhttp://https://www.youtube.com/watch?v=vrz0X07Rero\nConclusion vSAN Support Insight is a great initiative from VMware from Customer experience perspective which brings in a lot of advantage towards resolving customer problems and also help improve features and stability of the product , its highly recommended to have the CEIP program enabled if not enabled . Also make sure to proactively share the VC-UUID/Cluster-UUID while filing a support ticket to help resolve issues sooner . Most of the customer have already enabled CEIP and are greatly benefited with vSAN Support Insight .\n","link":"https://blogs.hareeshkg.bid/post/what-is-vsan-support-insight-ceip/","section":"post","tags":[],"title":"What is vSAN Support Insight – CEIP?"},{"body":"I have been asked about vSAN encryption a lot these days as the security requirement and standards are exponentially increasing in the IT industry . Data encryption is a key factor to make sure mission critical data are not stolen . Here is a quick and simple guide to configure vSAN-encryption with a Hy-Trust appliance and best practices to deploy and configure encryption cluster with High-Availability .\nIntroduction to vSAN Encryption Here is a pictorial representation of vSAN Encryption work flow which explains different hand shakes which are going to be executed until we finally encrypt the physical disks which are going to be used with vSAN , you may also refer to VMware-official-page for additional information .\nKEK retrieval process VC forms mutual trust with KMS server over SSL/TLS channel using KMIP protocol\nKMS certificate when uploaded to VC gets stored ins VC trusted cert store\nVC requests for KEK from KMS server\nKMS generates KEKID for vSAN cluster\nVC pushes KMS certificate to all the hosts part of vSAN cluster with KEKID\nHosts contact KMS server using KEKID and retrieve KEK\nCertificate store of hosts KMS certificate is /etc/vmware/ssl\nKMS configuration stores at /etc/vmware/esx.conf\nNote* : Once initial configuration is pushed down to the ESXi host which are going to participate in an encrypted vSAN cluster , there is no requirement for VC to keep the communication active between the KMS-Servers and ESXi hosts . VC is just required for the initial configuration .\nEncrypting data using DEK Host stores KEK in its RAM (safe area with in RAM Key Cache)\nKEK is used to encrypt DEK for all the vSAN disks which is generated randomly\nDEK temporarily gets stored in host key cache with KEK to encrypt the data in plain text\nKey cache is used to safeguard KEK \u0026amp; DEK from unauthorized access\nLSOM is the component which retrieves the DEK from host memory and encrypts \u0026amp; decrypts the data and then place the data on disks\nKEK eventually encrypts the DEK and DEK gets stored permanently somewhere in vSAN datastore\n** Host Key: **\nHost key is provided by KMS server per vSAN Cluster\nIn case of esxi host crash coredump will be generated, coredump will have KEK \u0026amp; DEK as we know host stores KEK \u0026amp; DEK in RAM\nTo protect unauthorized access, host key is used to encrypt coredump\nSummary In case of a compromising situation We can get the KEK keys and DEK keys recreated thru vcenter , however is an exhaustive and time intensive process\nSteps to deploy and configure a KMS HA cluster with Hy-Trust-KMS In this deployment we are going to look at configuring a **HyTrust KeyControl **virtual appliance version 4.2 with a vSAN cluster running vSAN 6.7 , same procedure applies for a vSAN 6.6.x cluster as well .\nPre-Requisite :\nMake sure you deploy KMS node on non-encrypted storage , definitely not on the same Cluster which we are about to encrypt data , which will lead to a catch-22 situation.\nVerify that the systems you want to use meet the basic system requirements\nConfigure the first KeyControl node\nInitialize KeyControl through the KeyControl webGUI for the first node\nIf desired, install additional KeyControl nodes and join them to the cluster. The number of nodes you can install is dictated by your KeyControl license , always a good practice to have at least two appliance in HA configuration . VCenter allows to add up to 6 appliance on a single KMS-Cluster .\nStart Deploying HyTrust KeyControl 4.2 OVA Fill up OVA deployment parameters ⇒ complete the deployment ⇒ power on the VM ⇒ Complete initial configuration on the Console 2. Head over to the webpage , then login with user/password as “secroot/secroot” . Remember this is not the same password which was setup on the console and will needs to be changed after first time login on the web page .\nAccept all agreements and setup a new web-GUI password . You may choose to configure Email alerts to receive alerts from the KMS-appliance , I chose to skip this . Enable KMIP on the KMS server and hit proceed to continue with Overwrite if prompted as we are configuring this KMS cluster for the first time. Also make a note of the port number . Create client side certificate for VC to be able to communicate with the KMS server . Add the KMS server to the vcenter and authenticate communication between the KMS certificate and vcenter server . Hytrust uses KMS certificate and private key upload method to the vcenter server for KMS to trust the vcenter server . Hence we will need to download the client certificate (KMS-certificate and private key) and upload it to the vcenter server . Note: PEM file which is obtained from KMS contains both certs which are required . Same file is used for KMS and private key section .\nDeploy the secondary KMS appliance for HA-clustering Create a 16 character passphrase and authenticate connectivity from the web-GUI on the primary node . Add the secondary node to the existing KMS-cluster. Configure-vSAN cluster with Encryption Enabled[ ](/images/how-to-configure-vsan-encryption-crypt_48.jpg) [root@enc-h1:~] vdq -iH Mappings: DiskMapping[0]: SSD: naa.6000c295967afd2c6e6f2a7a862bf998 MD: naa.6000c29b928af4bfa3fc109492da454e\n[root@enc-h1:] localcli vsan storage list |grep -i 'Device|In CMMDS|VSAN UUID|VSAN Disk Group UUID|Is Capacity Tier' |sed 'N;N;N;;N;s/\\n//g'; Device: naa.6000c29b928af4bfa3fc109492da454e VSAN UUID: 5218a577-d999-0cbc-a084-33d29473cf79 VSAN Disk Group UUID: 524e53e9-7e08-30e6-61c0-017bc8a01ad0 In CMMDS: true Is Capacity Tier: true Device: naa.6000c295967afd2c6e6f2a7a862bf998 VSAN UUID: 524e53e9-7e08-30e6-61c0-017bc8a01ad0 VSAN Disk Group UUID: 524e53e9-7e08-30e6-61c0-017bc8a01ad0 In CMMDS: true Is Capacity Tier: false [root@enc-h1:] localcli vsan storage list |grep -i encryption Encryption: true Encryption: true\nHost Live vmkernel.log during DG creation :\n2018-08-24T02:30:29.123Z cpu0:2099983 opID=25e20366)Encryption enabled. 2018-08-24T02:30:29.123Z cpu0:2099983 opID=25e20366)StorageEfficiency enabled. 2018-08-24T02:30:29.123Z cpu0:2099983 opID=25e20366)Config: 862: \u0026quot;LicensedFeatures\u0026quot; = \u0026quot;vit,allflash,stretchedcluster,erasurecoding,storageefficiency,encryption\u0026quot;, Old value: \u0026quot;allflash,stretchedcluster\u0026quot; (Status: 0x0) 2018-08-24T02:36:51.221Z cpu0:2110224)CpuSched: 693: user latency of 2112402 VSAN_0x43045b04df28_PLOG 0 changed by 2110224 vsanmgmtd-worke -6 2018-08-24T02:36:51.281Z cpu0:2110224)Created VSAN Slab LSOM_IORETRY_EncSlab (objSize=65536 align=64 minObj=0 maxObj=819 overheadObj=0 minMemUsage=0k maxMemUsage=55692k) 2018-08-24T02:36:51.314Z cpu0:2110224)LSOMCommon: IORETRY_Create:2570: An IORETRY queue for diskUUID 524e53e9-7e08-30e6-61c0-017bc8a01ad0 (0x4309dd9ac200) is encrypted 2018-08-24T02:36:51.328Z cpu0:2110224)LSOMCommon: IORETRY_Create:2585: An IORETRY queue for diskUUID 524e53e9-7e08-30e6-61c0-017bc8a01ad0 (0x4309dd9b9a90) is NOT encrypted 2018-08-24T02:36:52.331Z cpu0:2110224)PLOG: PLOGAnnounceSSD:7268: Successfully added VSAN SSD (naa.6000c295967afd2c6e6f2a7a862bf998:2) with UUID 524e53e9-7e08-30e6-61c0-017bc8a01ad0. kt 0, en 1, enC 1. 2018-08-24T02:36:52.331Z cpu1:2112402)CpuSched: 693: user latency of 2112411 VSAN_0x43045b477e58_LSOMLLOG 0 changed by 2112402 VSAN_0x43045b04df28_PLOG -6 2018-08-24T02:36:52.339Z cpu0:2110224)LSOMCommon: IORETRY_Create:2570: An IORETRY queue for diskUUID 5218a577-d999-0cbc-a084-33d29473cf79 (0x4309de580610) is encrypted 2018-08-24T02:36:52.352Z cpu0:2110224)LSOMCommon: IORETRY_Create:2585: An IORETRY queue for diskUUID 5218a577-d999-0cbc-a084-33d29473cf79 (0x4309de6905d0) is NOT encrypted 2018-08-24T02:36:53.356Z cpu1:2110224)PLOG: PLOGInitAndAnnounceMD:7737: Successfully announced VSAN MD (naa.6000c29b928af4bfa3fc109492da454e:2) with UUID: 5218a577-d999-0cbc-a084-33d29473cf79. kt 0, en 1, enC 1.\nPlease feel free to read thru vSAN-Encryption Troublehsooting Deep-Dive blog to understand different scenarios and methods to approach and fix problems with KMS and Disk groups\n","link":"https://blogs.hareeshkg.bid/post/how-to-configure-vsan-encryption/","section":"post","tags":[],"title":"How to  Configure vSAN Encryption?"},{"body":"There are many questions revolving around different layers of vSAN back-end architecture and how we can isolate problems with vSAN or in general approach towards troubleshooting a problem with vSAN . Here is an article which can help you isolate and potentially resolve a problem which involves a vSAN environment by isolating the problem in different layers by breaking down architecture in a simple way.\nThe approach towards troubleshooting a problem can be categorized in 5 layers similar to OSI model of networking . Application layer , Management layer , Object layer , Object layer and physical layer . We further see examples on how we can tag/categorize issues with each layer , which will help you figure out the source of the problem and quickly finding a solution . You may not be able to find the solution to a problem at all times however be able to categorize the issue under one of these layers , being able to explain the issue accurately to the Technical Support team can help them quickly narrow down the problem and fix that .\nThis PNOMA architecture includes certain critical components of vSAN which helps to isolate and troubleshoot vSAN , however there few other components we revolve around these and are dependent on one or more of these components .\nApplication Framework This frame work consists of certain key elements like the vcenter server (VPXD,vsan health etc) , VPXA and Hostd services . These services are categorized in the application layer as all interactions between users and vSAN backend components happen over this layer either thru hosts directly or thru vcenter server . For example virtual machine creation/vmdk creation or deletion , storage policy creation/modification, enabling vSAN features and services , vSAN health monitoring , DG creation , FD creation .. etc . All these tasks are performed by the users primarily thru vcenter server either from the webclient or the RVC (Ruby Vsphere Console) , these tasks hence translate to actions at the back-end on individual hosts thru vCenter API to VPXA agent running on the hosts , the tasks are intern translated to actions on the HOSTD service (running on ESXi hosts) which starts to invoke required libraries to complete the task.\nManagement Framework Once a task is received at the host layer and if the task is related to vSAN the host invokes required libraries for vSAN and facilitates certain type of tasks , some of such libraries / daemons are DISKLIB , OSFSD VSANVPD and few others .\nWhy are we categorizing these services under management ? , These are the critical services that are needed for us to be able to create/modify and delete an object within vSAN and if any of these services does not function correctly we will not able to create/modify and delete objects .\nDISKLIB : Disklib job is to invoke disk creation depending on the type of disk , this can be on a VMFS , NFS , vSAN or a vVOL depending on the type of datastore we choose , since we are discussing vSAN here this will invoke a vSAN object creation (can be a vdisk/namespace/vswap/vmem etc)\nOSFSD or OSFS-Daemon is responsible for the object creation/query task within the vSAN filesystem .\nvSANVPD or vSAN VASA Provider Daemon : vSAN uses vsanvpd service to expose its feature of SPBM , RAID , fault-tolerance , object space reservation , striping ..etc . The ESXi hosts (Nodes) part of the vSAN cluster runs the VASA provider and exposes this to the vCenter server over port 8080 so that vCenter can understand all the features and capabilities of vSAN . If the vSANVPD services are down you will not be able to create new VMs or change policies for existing virtual machines . See Troubleshooting Guide for vSAN VASA providers\nObject Framework The object framework is all about object life cycle within the vSAN filesystem . The key components involved in a vSAN life-cycle are DOM , CLOM and CMMDS . These components are responsible to create the objects with a specific configuration defined thru SPBM , validate the configuration if it can actually be satisfied or not , once the object is created within vSAN . It has to update all the host about the object type , owner of the object , policy/layout of the object , child components associated with it to all the hosts .\nDistributed Object Manager (DOM): The DOM is responsible for creating the components and distribute them across the cluster. Once a DOM-object is created one of the nodes (Host) will be nominated as the DOM owner for that object and this host will be responsible to handle all IOPS to that DOM-Object (Ex : vdisk, snapshot, vmnamespace, vmswap, vmem..etc) by locating the respective child components across cluster and redirecting the IO to respective components over vSAN network\nCluster Monitoring, Membership and Directory Services (CMMDS): The purpose of CMMDS is to discover and maintain the vSAN cluster. It stores metadata information such as policies, and RAID configuration for all objects within vSAN.\nCluster Level Object Manager (CLOM): Given a storage policy, checks to see if there are enough disk groups to satisfy that policy. CLOM is the brain which decides on the components and witnesses that need to be created and where they need to be placed in a cluster.\nNetwork Framework Reliable Datagram Transport(RDT): RDT is the protocol used by vsan for communication between hosts over the vSAN vmkernel ports (cmmds , I/O flow etc). It is optimized to send very large files. Physical Layer This layer is where data traverses and reside within a vSAN file-system hence the name physical layer. Important components which constitutes the physical layer are LSOM , Cache Tier drives and Capacity tier drives .\nLocal Log-structured Object Manager(LSOM): The LSOM is responsible for locally storing the data on vSAN file system as vSAN Component or LSOM-Object (data component / witness component). These objects are created on top of the capacity tier drives depending on the geometry size advised by CLOM . This also includes PLOG and LLOG which stores the metadata for vSAN Virsto file-system and de-dupe metadata information .\nCache Tier drive : As the name suggests the sole purpose of this drive is to help I/O traverse faster within vSAN , these drives are generally faster and higher endurance SSD/NVMe drives which are used to cache reads and writes in a hybrid vSAN cluster and is dedicated for writes in a All-Flash environment .\nCapacity Tier drive : These drive store LSOM Objects created within the vSAN filesystem , they directly service reads in an All-Flash environment. When De-dup\u0026amp;compression is executed while the data gets de-staged from cache tier to capacity .\nI am planning to write some articles in the near future by illustrating different issues and categorize them under PNOMA which should give us better understanding . Please make sure to follow the blog for more content .\n","link":"https://blogs.hareeshkg.bid/post/pnoma-a-vsan-troubleshooting-framework/","section":"post","tags":[],"title":"PNOMA – A vSAN Troubleshooting Framework"},{"body":"New Blog Post for “How to install , configure and Migrate to SRM 8.2” ,if you are interested to deploy SRM 8.2 instead of SRM 8.1 . SRM 8.2 comes with the appliance edition and is far more easier and faster to deploy and configure . This was just released in May 2019 .\nSite recovery Manager 8.1 and vSphere Replication 8.1 is available out in the market for quite sometime . Here is my First impressions and a simple guide to deploy -configure Site recovery Manager and vSphere replication 8.1 , comparison with older version and conclusion .\nPre-Requisite Two vCenter servers (embedded/external) running vpshere 6.0 U3/6.5/6.7\nA windows server for each site to install SRM (If VC was running on a windows platform you may use the same server)\nSince this is a vSAN environment I have got 3 ESXi hosts on each site .\nOne VR-appliance on each site to be deployed .\nSRM windows boxes can be outside of the clusters however vsphere replication appliance must always run within the same environment .\nSRM install is pretty much a straight forward deployment , I used the embedded vPostgres database for both sites and vsphere replication were configured with embedded vPostgres database.\nMy setup Two vcenter servers running with external platform service controllers version 6.7 GA with SSO multi-site linked mode\n3 node nested-ESXi all flash vSAN cluster running 6.7GA\nOne VR appliance deployed on each site .\nChallenges VR appliance OVF deployment is not supported thru the HTML(Clarity UI) for vcenter server , you will have to ONLY deploy this thru the Flash based webclient . Deployment thru the hosts directly is also un-supported .\nMake sure PSC,VC,SRM and ESXI hosts machines are syncing the time with a common NTP servers , otherwise your SRM installation or Vsphere replication registration with VC will have issues\nFirst Impressions I am very happy to see that we have finally decoupled the SRM and vsphere replication appliance dependency with the version of vcenter servers and ESXi hosts between both sites .\nWe also have lost the dependency of webclient service on the vcenter server as the UI is now native with the SRM and vsphere replication appliance with the all new Clarity UI .\nvSphere replication Management and Sit recovery is now combined together in a single view . This can be launched with SRM or vSphere replication appliance “IP-Address/UI”\nDeploy-Configure and Manage As explained above , my setup is configured with between two vcenter servers running vSAN/vSphere 6.7 with external platform service controllers in a multi-site configuration within the same SSO Domain.\nDeployed PSC one first primary site and one for the secondary site within the same SSO domain but different site names .\nLater deployed one vcenter server for each site and pointed them to the respective PSC.\nInstalled SRM on two different server 2012 here in my case and pointed each SRM to the respective sites . I was later able to see the “Site Recovery” plujgin showing up on the webclient and when clicked launches a new window , which is totally different from previous versions where all site recovery actions were controlled within the same webclient . This plugin shows up on both HTML-UI and Flash based UI . I liked to use this within HMTL client which is lot faster .\nDeploy the vsphere replication appliance thru the Flash based webclient on both sites .\nPower on the appliance and head to the VAMI interface-page-UI which https://VR-Appliance-IP:548- for both sites , configure and register the VR to their respective sites these steps remains the same as previous versions . Once both VR are registered with the respecrtive PSC/VC we should see both SRM and VR in OK state for both sites .\nWhen we click on Open site recovery a new tab opens with the IP address of the vSphere-replication appliance-IP/UI for the respective site . I also found that the UI remains the same if we open up the IP address of SRM-server/UI , there is no difference between the UI as demonstrated below . Here the IP address 192.168.105.3 is for the SRM and 192.168.105.4 is for the vSphere-Replication appliance at the primary-site and both shows the same UI as they interlinked with version 8.1 we no longer manage them separately .\nNow we will need to pair the two vsphere replication appliance between two sites after which we should see additional options to replicate virtual machines and later on we should see options to configure protection groups and recovery plans within the same UI .\nPerform initial site paring which pretty much straight forward . Please note while pairing the two sites you will note both VR and SRM service will be detected at scondary site and you can pair them in the same window .\nThe task will take about a minute to complete the Site-Pairing we see the pairing task for both SRM and vSphere Replication . We can now hit the view details button and we land at the Summary section which shows the status of Site pairs for SRM and vsphere replication and their health status. Configure replication for one of the virtual machine , here I have just used a nested ESXI OVF template called “Nested-replica-test” on the primary site for test purpose and we shall see how replication is now configured with new UI which is totally different from how it was done in previous versions . When we right-click the VM ⇒ SIte Recovery ⇒ Configure replication , it now pops-up a new tab and the configuration for vsphere-replication is continued thru the new-UI for SRM/VR . Here we have to just follow the wizard by choosing the Target-site , datastore, storage policy at target site , RPO value which allows as low as 5 mins (earlier version only support 15 mins or higher this is new) , we can also enable network compression . We are allowed to even create the protection group or add to an existing one which will make our life so much easy as we can save a lot of time to create or add the VMs to a protection group from site recovery menu in previous versions . Finally the replication status will now reflect under replications tab from the new-UI . Please note that if we donot create/ enable a dedicated vSphere Replication vmkernal traffic the replication-servers use the management kernel for performing replication between sites . It is recommended to have a dedicated vlan for networking with dedicated vmkernel for larger environments .\nConfiguring Protection group from New-UI , the wizard looks almost same as the older predecessors , however this also includes creation of recovery plan while creating the protection group I chose to create the RP along with protection group which different from previous versions .\nWe ran into an issue with protection group “Unanle to protect VM due to unresolved devices “, which is expected as we have no mappings for network , folders , storage , resource pools, storage policy and place holder datastore from SRM perspective between the two sites . We should see the PG go back to OK state once this is successfully configured . Simply proceed to create the mappings for all the parameter successfully . Now right-click the PG and hit “Configure-All VMs” we should see the protection status go back to OK state and the place-holder created at DR site. What has changed about the vSphere-Replication from the host perspective . Let check the replication status over SSH and see if there are new commands available for vSphere replication from host perspective . Based on the below screenshot the commands from the host perspective still remains same from the previous version , if the vcenter server is down we will still be able to start/stop/pause/reconfigure replication at times if the two replication servers were still authenticated with each other .\nPerform IP customization settings on the new interface is same as older versions . Navigate to the Recovery Plan ⇒ Select a Recovery Plan ⇒ Select Virtual Machines ⇒ Configure Recovery ⇒ Choose Required IP Customization options entry values in required sections . Run a test/planned Migration . Running a planned Migration test is no different from previous versions . Steps involved are pretty much similar to previous versions , latest data is synced between two sites ⇒ powers off the production site VM ⇒At the DR site the place holder VM is reconfigured ⇒ Reloads with the new configuration and attaches the vmdk ⇒ VM is powered ⇒ IP Customization script is run ⇒ VM reboots and comes with the new IP address . Run the re-protect to reverse the Direction of protection and replicate the data back to primary site . Conclusions The new UI separation form webclient is a very good move . Webclient will have a huge relief , we will no longer see the 120 second timeout issue while working with SRM and vSphere replication\nWe are allowed to perform actions on vsphere replication or SRM without having to log in to webclient .\nReally loved the integration between VR and SRM on the same UI , allowing us to pair VR and SRM on the same wizard , create protection group and recovery plan while creating replication task . This will save huge amount of time .\nThis is a major break through for Users having different version of hosts/VC in primary site and DR site . Upgrading ESXI and VC is not going to affect/disrupt vsphere replication or SRM and viceversa .\nMinimum RPO time reduced to 5 mins from 15 mins which is a great move .\nPlease also check out Reconfigure Replication using existing seeds . Please also take a look at embedded deployment considerations which will be out soon.\n","link":"https://blogs.hareeshkg.bid/post/how-to-install-and-configure-site-recovery-81/","section":"post","tags":[],"title":"How to install and configure Site Recovery 8.1?"},{"body":"I highly encourage you to check out my previous blog to understand how vSAN encryption works before you jump into troubleshooting problems with vSAN and a KMS server .\nThis blog should give you a general approach towards troubleshooting problems with vSAN-encryption and also be able to look at a specific log location in order to isolate and fix a specific problem . I have also included certain test-case scenario to see how the logging looks on the host when there is an issue with vSAN-encryption .\nPreliminary checklist Content for checklist is available here at : Checklist Check KMS connectivity As discussed in the previous introduction blog , it is highly critical to note that the KMS server should NOT be deployed on the same cluster which is being encrypted which will lead to a catch-22 situation . Take a look the file **“esx.conf” **which should be under /etc/vmware/ and grep kmip to view KMS configuration on the ESXi host if the KMS was configured with FQDN , then it is necessary for the host to be able to resolve the KMS server thru its FQDN . It is okay to have VC down in all scenario as long as the KMS is reachable for the host .\n[root@enc-h1:] grep kmip /etc/vmware/esx.conf /vsan/kmipServer/child[0001]/old = \u0026quot;false\u0026quot; /vsan/kmipServer/child[0001]/port = \u0026quot;5696\u0026quot; /vsan/kmipServer/child[0001]/address = \u0026quot;hyt-s.vsensei.local\u0026quot; /vsan/kmipServer/child[0001]/name = \u0026quot;hyt-Secondary\u0026quot; /vsan/kmipServer/child[0001]/kmipClusterId = \u0026quot;vSAN-Encryption\u0026quot; /vsan/kmipServer/child[0001]/kmskey = \u0026quot;vSAN-Encryption/hyt-Secondary\u0026quot; /vsan/kmipServer/child[0000]/kmskey = \u0026quot;vSAN-Encryption/hyt-primary\u0026quot; /vsan/kmipServer/child[0000]/kmipClusterId = \u0026quot;vSAN-Encryption\u0026quot; /vsan/kmipServer/child[0000]/name = \u0026quot;hyt-primary\u0026quot; /vsan/kmipServer/child[0000]/address = \u0026quot;hyt-p.vsensei.local\u0026quot; /vsan/kmipServer/child[0000]/port = \u0026quot;5696\u0026quot; /vsan/kmipServer/child[0000]/old = \u0026quot;false\u0026quot; /vsan/kmipClusterId = \u0026quot;vSAN-Encryption\u0026quot; [root@enc-h1:]\nIn the above output we see that we have two KMS server configuration pushed from VC to Hosts which are “hyt-p.vsensei.local” and “hyt-s.vsensei.local” and port used for connecting to KMS is 5696 . Confirm if we have the right KMIP certificates Navigate to the path /etc/vmware/ssl on one of the hosts which is part of the vSAN-encrypted cluster and check the certificates on the files vsan_kms_client.crt (client Certificate) , vsan_kms_client.key (private key)and vsan_kms_castore.pem . The vsan_kms_castore.pem file is a copy of the server certificate that the host uses to compare with the cert returned by the KMIP server during initial SSL handshake. If the server cert has been changed and does not match what ESXi has stored here, the connection will not be established.\nHere is how you can compare and confirm if we have the right certificates and keys on the respective files we have on the hosts .\n[root@enc-h1:/etc/vmware/ssl] ls -la kms -rw-r--r-T 1 root root 2726 Aug 24 02:30 vsan_kms_castore.pem -rw-r--r-T 1 root root 0 Aug 24 02:38 vsan_kms_castore_old.pem -rw-r--r-T 1 root root 1302 Aug 24 02:30 vsan_kms_client.crt -rw-r--r-T 1 root root 1708 Aug 24 02:30 vsan_kms_client.key -rw-r--r-T 1 root root 0 Aug 24 02:38 vsan_kms_client_old.crt -rw-r--r-T 1 root root 0 Aug 24 02:38 vsan_kms_client_old.key\nFrom , the above list of files , the first file we will need to look at is “***vsan_kms_client.crt***” this KMS client key should match the KMS-Client-Certificate which was issued by the KMS server to the vcenter server , if you dont have the file locally to compare the key you may choose to re download pem file from the KMS server . Now cat the file “vsan_kms_client.crt” on the ESXI host and compare it with first half of the file which contains client certificate here its “vcenter.pem” which we downloaded from the KMS server .\nCat the next file “vsan_kms_client.key” , this should match with the private key on the same PEM file which was downloaded .\nNext up is “vsan_kms_castore.pem” , this file contains server side certificate from the KMS server , the contents /certificates in this file depends on the total number of KMS servers with which the ESXi hosts are connected , in my current example here we have two KMS servers (primary and secondary ) , hence we should have two certs one from each KMS server .\nLog in to Hy-Trust (KMS used in my lab) ⇒ Clusters⇒ Select-first-KMS⇒ View Current Certificate , this certificate must match the content on one of the certificates within “vsan_kms_castore.pem” file , similarly the other certificate on the same file must match the server certificate on the secondary KMS server .\nFrom the above screenshots we can see that all certificates stored on the hosts on respective files matches certificates issued/stored on the KMS servers in question . If vCenter is available and the host is missing any of this information, vCenter will provide the host with copies of the certificates it has stored in VECS. If the vcenter server is down we should be able to manually copy the contents from the KMS server on to those respective files which should fix/resolve certificate mismatches on the hosts .\nEnsure that host can enter crypto-safe mode See content for details around host crypto safe mode here\nTo determine if a HostKey has been installed (i.e. the host is crypto-safe), you can use the UI (if available).Select the host in the inventory and go to Configure ⇒Security Profile ⇒Host Encryption Mode\nAll additional details from krypto-safemode is avaialable here\nCertificate related issues If we were missing any of the file which contains certificate data under /etc/vmware/ssl (vsan_kms_castore.pem , vsan_kms_client.crt , vsan_kms_client.key ) you will see errors relating to certificates on the same syslog.log file . To simulate this problem I simply placed host-1 in MM Mode and deleted the file **vsan_kms_castore.pem **under /etc/vmware/ssl and rebooted the host .\nAfter I rebooted the host , we see that the file “vsan_kms_castore.pem” was recreated , however was of 0 bytes with content inside it . Hence the Disk group on this did not mount over the reboot , lets check the logs for error messages .\nFrom syslog.log :\n2018-08-25T14:38:17Z jumpstart[2097463]: VsanUtil: Get kms client key and cert, old:0 2018-08-25T14:38:17Z jumpstart[2097463]: VsanUtil: GetKmsServerCerts KMS certs not found 2018-08-25T14:38:17Z jumpstart[2097463]: VsanInfoImpl: Joining vSAN cluster 5232826c-507e-1f30-197d-605f6d44d413 2018-08-25T14:38:17Z jumpstart[2097463]: VsanInfoImpl: SyncConfigurationCallback called 2018-08-25T14:38:17Z jumpstart[2097463]: VsanSysinfo: Loading module cmmds 2018-08-25T14:38:22Z jumpstart[2097463]: VsanInfoImpl: Retrieving the host key with keyId: b2385c26-a745-11e8-bbd4-005056b85466 2018-08-25T14:38:22Z jumpstart[2097463]: VsanUtil: Get kms client key and cert, old:1 2018-08-25T14:38:22Z jumpstart[2097463]: VsanUtil: GetKmsServerCerts Old KMS certs not found 2018-08-25T14:38:22Z jumpstart[2097463]: VsanInfoImpl: Failed to retrieve host key from KMS: Invalid key or certs. Will have 1 retries. 2018-08-25T14:38:27Z jumpstart[2097463]: VsanUtil: Get kms client key and cert, old:1 2018-08-25T14:38:27Z jumpstart[2097463]: VsanUtil: GetKmsServerCerts Old KMS certs not found 2018-08-25T14:38:27Z jumpstart[2097463]: VsanInfoImpl: Failed to retrieve host key from KMS: Invalid key or certs. Will have 0 retries. 2018-08-25T14:38:27Z jumpstart[2097463]: VsanInfoImpl: Failed to load DEKs: Invalid key or certs\n*From Boot.gz file post reboot , we see that the disks were **NOT announced **post reboot as we could not retrieve the right key file from host cache as host cache itself could not load the keys from “**vsan_kms_castore.pem**“* [root@enc-h1:/var/log] vdq -i [ { \u0026quot;SSD\u0026quot; : \u0026quot;naa.6000c295967afd2c6e6f2a7a862bf998\u0026quot;, \u0026quot;MD\u0026quot; : [ \u0026quot;naa.6000c29b928af4bfa3fc109492da454e\u0026quot;, ] }, ] [root@enc-h1:/var/log] zcat boot.gz | grep -i \u0026quot;naa.6000c295967afd2c6e6f2a7a862bf998\u0026quot;\n2018-08-25T14:38:31.474Z cpu1:2097464)PLOG: PLOG_InitDevice:276: Initialized device M naa.6000c295967afd2c6e6f2a7a862bf998:1 0x430abd085fb0 quiesceTask 0x430abd087230 on SSD 524e53e9-7e08-30e6-61c0-017bc8a01ad0deviceUUID 00000000-0000-0000-0000-00000000$ 2018-08-25T14:38:31.474Z cpu1:2097464)WARNING: PLOG: PLOGValidateDisk:2867: Not announcing disk naa.6000c295967afd2c6e6f2a7a862bf998:1 until key is available. Failed obtaining it with status Not found. 2018-08-25T14:38:37.244Z cpu1:2097464)PLOG: PLOGProbeDevice:5728: Probed plog device \u0026lt;naa.6000c295967afd2c6e6f2a7a862bf998:2\u0026gt; 0x430abd0844d0 exists.. continue with old entry 2018-08-25T14:38:37.247Z cpu1:2097464)PLOG: PLOGProbeDevice:5728: Probed plog device \u0026lt;naa.6000c295967afd2c6e6f2a7a862bf998:1\u0026gt; 0x430abd085fb0 exists.. continue with old entry 2018-08-25T14:38:37.247Z cpu1:2097464)WARNING: PLOG: PLOGValidateDisk:2867: Not announcing disk naa.6000c295967afd2c6e6f2a7a862bf998:1 until key is available. Failed obtaining it with status Not found.\n[root@enc-h1:/var/log] zcat boot.gz | grep -i \u0026quot;naa.6000c29b928af4bfa3fc109492da454e\u0026quot;\n2018-08-25T14:38:31.475Z cpu1:2097464)PLOG: PLOG_InitDevice:276: Initialized device M naa.6000c29b928af4bfa3fc109492da454e:1 0x430abd087720 quiesceTask 0x430abd0889a0 on SSD 524e53e9-7e08-30e6-61c0-017bc8a01ad0deviceUUID 00000000-0000-0000-0000-00000000$ 2018-08-25T14:38:31.475Z cpu1:2097464)WARNING: PLOG: PLOGValidateDisk:2867: Not announcing disk naa.6000c29b928af4bfa3fc109492da454e:1 until key is available. Failed obtaining it with status Not found. 2018-08-25T14:38:31.475Z cpu1:2097464)PLOG: PLOG_InitDevice:276: Initialized device D naa.6000c29b928af4bfa3fc109492da454e:2 0x430abd088ef0 quiesceTask 0x430abd08a170 on SSD 524e53e9-7e08-30e6-61c0-017bc8a01ad0deviceUUID 00000000-0000-0000-0000-00000000$ 2018-08-25T14:38:37.248Z cpu1:2097464)PLOG: PLOGProbeDevice:5728: Probed plog device \u0026lt;naa.6000c29b928af4bfa3fc109492da454e:1\u0026gt; 0x430abd087720 exists.. continue with old entry 2018-08-25T14:38:37.248Z cpu1:2097464)WARNING: PLOG: PLOGValidateDisk:2867: Not announcing disk naa.6000c29b928af4bfa3fc109492da454e:1 until key is available. Failed obtaining it with status Not found. 2018-08-25T14:38:37.248Z cpu1:2097464)PLOG: PLOGProbeDevice:5728: Probed plog device \u0026lt;naa.6000c29b928af4bfa3fc109492da454e:2\u0026gt; 0x430abd088ef0 exists.. continue with old entry [root@enc-h1:/var/log]\n*Next , I tried couple of things , which worked successfully . I had a copy of the key on a notepad , I could simply write to the same file , restarted the management agents and it successfully remounted the disk group , otherwise we can download the same file from a working host copy to the affected host and restart all the management services to get the DG mounted . To make this even simpler as I still had the vcenter in a managed state , I moved the host out of the cluster and moved it back , VC successfully replaced the file and restarting management agents got the DG mounted .* Note*: Moving the host out of the cluster and moving it back only works if VC was healthy and had full control over the affected host , i.e “vCenter state is authoritative” on the host . Disconnecting the host moving it out of the cluster and moving it back will not replace the file .\nOnce Host moved back in you will notice recent tasks , which updated the vSAN configuration on the all the host including “enc-h1.vsensei.local” which was moved back in . The file gets successfully replaced and disk group gets mounted successfully just after moving it back into cluster , exiting MM mode will reflect correct size of the vSAN datastore .\n[root@enc-h1:~] esxcli vsan storage list | grep -i cmmds In CMMDS: true In CMMDS: true\n1[root@enc-h1:~] df -h (Before Exiting MM Mode) 2 3Filesystem Size Used Available Use% Mounted on 4vsan 24.0G 2.4G 21.6G 10% /vmfs/volumes/vsanDatastore 5 6[root@enc-h1:~] df -h (After exiting MM Mode ) 7 8Filesystem Size Used Available Use% Mounted on 9vsan 32.0G 3.2G 28.8G 10% /vmfs/volumes/vsanDatastore Note* : In certain conditions even if the file is recreated and moved back in , the DiskGroup may not be mounted , a manual DG mount after restarting services also may not help . Its good to reboot the host and disks should get announced successfully post boot up . KEY error codes to note : Error code Meaning QLC_ERR_COMMUNICATE Failed to communicate to Key Management Server QLC_NEED_AUTH User authencication failed to Key Management Server QLC_ERR_TIMEOUT_EXPIRED Connection timeout to Key Management Server Please Make sure to read thru : https://thevirtualpaddy.wordpress.com/2018/07/11/troubleshooting-vsan-encryption-and-kms-connectivity/ for all additional troubleshooting sections for vSAN and VM encryption\n","link":"https://blogs.hareeshkg.bid/post/troubleshooting-vsan-encryption-deep-dive/","section":"post","tags":[],"title":"Troubleshooting vSAN Encryption Deep Dive"},{"body":"At many times we may see issues with webclient reporting a 120 second time out and we are unable to manage vSAN from the configure (Cluster ⇒ Configure ⇒ vSAN ) and monitor (Cluster ⇒ Monitor⇒ vSAN ) for vSAN . However from the SSH the cluster is health with no issues . This is generally indication of problems with the vSAN health services and some times it is the storage provider service which runs at the vcenter level which is responsible to poll and display information on the web-client for vSAN-UI area .\nGeneral issues seen when we have a problem with vSAN-health Plugin .\nUnable to create new cluster and enable vSAN services.\nUnable to view and Manage existing vSAN cluster from webclient .\nUnable to create and delete vSAN Disk groups from Webclient .\nThe vSAN health plugin reports no data .\nCannot enable / disable stretch clusters under Configure(Manage) TAB on a vSAN enabled Cluster .\nError : Unexpected status code : 503\nError : Error occurred while loading general configuration of vSAN\nError : The query execution timed out because of back-end property provider?\nHow to handle and troubleshoot issues with vSAN-Health Service Best way to isolate the problem is by checking if the behavior is only with one cluster on the vCenter server of if this is the same behavior with all clusters in the vCenter server . Once we find out the source of the problem if its one cluster in the vCenter server or if its seen with all cluster in the vCenter .\nHere is an example where we have manually stopped the vSAN health service on a vCenter server running 6.0 Update 3 latest patch . We clearly see that the vSAN health plugin shows no data , even the vSAN General , Disk Management and Health and performance service doesnot show any information .I also created a new cluster and later tried to enable vSAN , however we dont see the option to turn on vSAN . These set off issues clearly indicates problem with the health service plugin on the vCenter server (VMware-vsan-health service) which is responsible for these issues .\nNote** : we will not be seeing all of these problems with vSAN 6.6.1 (running 6.5U2) it looks like the dependency for the Configure -\u0026gt; vSAN tab is now removed , we can still see disk groups , create new clusters and enable vSAN on this release .\nStep1: A simple fix is to check if the vSAN-health service is running or not on vCenter server , if found to be stopped start the service manually and check back on the web-client to see if everything is back to normal . Check if the vsan-health / vmware-vsan-health is running on the vCenter server.\nVCSA Appliance vCenter-P:~ # service-control --status Running: vmware-cis-license (VMware License Service) vmware-eam (VMware ESX Agent Manager) vmware-invsvc (VMware Inventory Service) vmware-perfcharts (VMware Performance Charts) vmware-psc-client (VMware Platform Services Controller Client) vmware-rhttpproxy (VMware HTTP Reverse Proxy) vmware-sca (VMware Service Control Agent) vmware-sps (VMware vSphere Profile-Driven Storage Service) vmware-syslog (VMware Common Logging Service) vmware-syslog-health (VMware Syslog Health Service) vmware-vapi-endpoint (VMware vAPI Endpoint) vmware-vdcs (VMware Content Library Service) vmware-vpostgres (VMware Postgres) vmware-vpx-workflow (VMware vCenter Workflow Manager) vmware-vpxd (VMware vCenter Server) vmware-vsm (VMware vService Manager) vmware-vws (VMware System and Hardware Health Manager) vsphere-client ()INFO:root:\nStopped: vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vsan-health (VMware VSAN Health Service)Stopped: vmware-mbcs (VMware Message Bus Configuration Service) vmware-netdumper (VMware vSphere ESXi Dump Collector) vmware-rbd-watchdog (VMware vSphere Auto Deploy Waiter) vmware-vsan-health (VMware VSAN Health Service)\nStart the service by running : vCenter-P:~ # service-control --start vmware-vsan-health INFO:root:Service: vmware-vsan-health, Action: start Service: vmware-vsan-health, Action: start 2018-05-09T21:48:47.624Z Running command: ['/sbin/chkconfig', u'vmware-vsan-health'] 2018-05-09T21:48:47.676Z Done running command 2018-05-09T21:48:47.676Z Running command: ['/sbin/service', u'vmware-vsan-health', 'status'] 2018-05-09T21:48:47.685Z Done running command 2018-05-09T21:48:47.685Z Running command: ['/sbin/chkconfig', '--force', u'vmware-vsan-health', 'on'] 2018-05-09T21:48:47.731Z Done running command 2018-05-09T21:48:47.732Z Running command: ['/sbin/service', u'vmware-vsan-health', 'start'] 2018-05-09T21:48:47.746Z Done running command 2018-05-09T21:48:47.746Z Successfully started service vmware-vsan-health vCenter-P:~ #\n**Windows vCenter server **Check Status:\n[ ](/images/unable-to-manage-vsan-on-webclient-service-control-status.png)Start the Service :\nStep2: If this was the problem with just one cluster under the vCenter server and if other vSAN enabled cluster were responding fine . Then the problem is with one of the vSAN hosts in the cluster . Check if “vsanmgmt ” and vsanvpd service is running on all the hosts\nIf either of this service was found not to be running feel free to restart them , these services dont cause any issues to running production virtual machines as they are foe vSAN management and vSAN VASA provider service .\nEven if we found these services to be running fine on all the hosts , it is still okay to restart them and check back in the GUI , all GUI tasks should now work as expected .\n1[root@esxi01:~] /etc/init.d/vsanmgmtd status ; /etc/init.d/vsanvpd status 2vsanperfsvc is running 3vsanvpd is running. 4 5[root@esxi01:~] /etc/init.d/vsanmgmtd restart ; /etc/init.d/vsanvpd restart 6watchdog-vsanperfsvc: Terminating watchdog process with PID 67652 7vsanperfsvc started 8watchdog-vsanvpd: Terminating watchdog process with PID 67133 9vsanvpd stopped 10vsanvpd started Step3 : If the vSAN-health service wont startup . This is a very rare situation which can be caused due to some files missing or some files corrupted . We will need to investigate this by checking the logs for vSAN-health service . Note* : It is suggested to engage VMware support to further diagnose the error to fix the issue with health service crash after start and not recommended to troubleshoot this on your own .\nLogs to check :\nAppliance VCSA Path : /var/log/vmware/vsan-health/vmware-vsan-health-service.log\nWindowsVC Path : C:\\ProgramData\\VMware\\vCenterServer\\logs\\vsan-health\\vmware-vsan-health-service.log\n** **\nAdditional helpful KB : vSAN Health Service – HCL Health – Host issues retrieving hardware info\n","link":"https://blogs.hareeshkg.bid/post/unable-to-manage-vsan-on-webclient/","section":"post","tags":[],"title":"Unable to manage vSAN on Webclient"},{"body":"At many times we wonder why we are not able to re-apply policy on a vSAN object or a virtual machine , it may be grayed out and many VMs on the vSAN datastore shows up as non-compliant . Here is a complete guide which helps you understanding the role and importance of the VASA providers in a vSAN environment ,the difference between the latest vSAN 6.7 VP and previous versions and troubleshooting methods to fix VASA issues .\nIntroduction to VASA Providers VASA is shorthand for vSphere Storage APIs for Storage Awareness. When VASA was introduced back in vSphere 6.0 days it allowed storage arrays to integrate with vCenter for management functionality via server-side plug-ins or Vendor Providers. The storage provider exists on either the storage array service processor or it may also be a standalone host – this is at the discretion of the vendor. So what dis it do? VASA provider exposes the array features capabilities to the vCenter server and let vCenter take advantage of these features for a virtual machine life cycle or a datastore . In case of vSAN , vSAN also uses VASA provider to expose its feature of SPBM , RAID , fault-tolerance , object space reservation , striping ..etc . The ESXi hosts (Nodes) part of the cluster runs the VASA provider and exposes this to the vCenter server over port 8080 so that vCenter can understand all the features and capabilities of vSAN .\nVASA Provider with earlier version of vSAN In the earlier version of vSAN which includes 5.5,6.1,6.2 and 6.6 VASA providers were always exposed to the vCenter server thru port 8080 on each of the vSAN Nodes under the vCenter server . . Ideally when everything is fine we should see all hosts VASA providers listed under vCenter⇒Configure (*Manage)⇒Storage Providers as online and one of the providers for each cluster in an active state while rest will be reported as standby .If these providers were not listed correctly and if found not online/active we usually face some of the following issues\nUnable to provision new VMs on vSAN datastore .\nUnable to verify Storage compliance for any of the VMs in vSAN datastore.\nUnable to re-apply/apply new storage policy on any of the VMs under vSAN datastore\nAll existing VMs on vSAN datastore will show up as Non-Compliant .\nHow that we know the types of issues which we can face with VASA providers let see how we can troubleshoot it .\nStep1: Check if providers from all hosts are online and listed correctly . Login to the vCenter server , navigate vCenter⇒Configure (*Manage)⇒Storage Providers . filter the VASA provider list with key word vSAN and check if the providers from all hosts shows up online and at least on host should be in active .\n[ ](/images/how-to-troubleshoot-issues-with-vsan-storage-policy-compliance-and-vasa-providers-vasa-6.6-1.png)Step2: Re synchronize the vASA providers if we do not see VASA providers listed , if we still do not see the providers from all hosts , we will need to restart the VASA provider service for vSAN on all hosts and the do the resync once again .\nCheck if the vSAN VASA provider service status and if found not running restart it . We need to first SSH to all hosts in the affected cluster , check the vSAN provider status and restart it. Check the VASA provider status : [root@hostname:~] /etc/init.d/vsanvpd status vsanvpd is running.\nIf the vsanvpd is not running, start it manually 1[root@hostname:~] /etc/init.d/vsanvpd start 2 3vsanvpd started If the service is crashing due to some reason we can check the cause in vsanvpd.log. 1[root@hostname:~] cat /var/run/log/vsanvpd.log - If the VASA providers are running fine what next ? we need to check if the providers are reachable to vCenter server .To examine for port liveness, connect to the VASA provider via port 8080 and determine if the VASA XML information is returned. This process varies between the vCenter Server Appliance (VCSA) and Windows vCenter. **Appliance :** Use the ‘cURL’ utility to check the VASA Provider:\ncurl –insecure https://:8080/version.xml\n(example : root@vcsa1 [ ~ ]# curl –insecure https://esxi-2.gsslabs.org:8080/version.xml\n)\nWindows : Use a web browser to check the VASA Provider by navigating to ‘https://:8080/version.xml’\n**Check ESXi host firewalls: Examine the host “Security Profile” and ensure that the ‘vsanvp’ rule is enabled to permit host communication over port 8080.\n**\n**Check vCenter Server firewall **\nOn Windows vCenter Server, check that the Windows Firewall is either disabled, or that **all** VMware-installed rules are active. In addition, check for custom rules that may be interfering with port 8080 outbound or inbound. On the VCSA, the firewall should be correctly configured by default. - **Examine VASA certificates **If the VASA provider is running and it is reachable by vCenter Server, the problem may be related to certificates. VASA and SPBM use certificate exchange, and the vCenter Server must accept the VASA provider certificates.\nCertificate-related problems will be called in the SPBM Java process’s wrapper log. The location varies by vCenter Server type.\nWindows vCenter Server: %ProgramData%\\VMware\\vCenterServer\\logs\\vmware-sps\\wrapper.log VCSA: /var/log/vmware/vmware-sps/wrapper.log\nIn some cases the VASA provider certificate may have a 0 Byte size. To resolve this, the host-side (provider) VASA certificates are required, These are stored on each ESXi host in /etc/vmware/ssl/ [root@esxi-1:~] ls -lah /etc/vmware/ssl/ | grep -i vsanvp -r--r--r-T 1 root root 0 Apr 7 2017 .#vsanvp_castore.pem -rw-r--r-- 1 root root 3.1K May 6 06:03 vsanvp.pem -rw-r--r-- 1 root root 2.1K May 6 06:18 vsanvp_castore.pem -rw-rw-rw- 1 root root 64 May 6 12:30 vsanvp_secret\nWhat is new with VASA in vSAN 6.7 In vSAN 6.7 the VP does not run on all ESXi hosts as it did in previous versions of vSAN. The VP now runs on vCenter server, therefore the SPBM service and the VP both share the same certificate.\nThe new vSAN VP can self register to the vSAN VP side, if vSAN VP is offline or unregistered then we can register back to the SPBM side automatically\nThere is better integration with the health service and the API’s\nSPS now needs to support less as it only needs to support the IO filters\nVASA Provider uses port 8080 TCP Unicast for internal communication\nVASA does not need to run on the 6.5 hosts as it is running on the 6.7 VC\nHere is a comparision between the two version vSAN 6.6 and vSAN 6.7 . The VP in a vSAN 6.6 environment on the left and a vSAN 6.7 environment on the right. The 6.6 environment shows that the VP is being managed by esxi-1, however the 6.7 environment shows that the VP is being managed internally.When moving to 6.7 SMS will remove all host based VP’s. Once upgrade is complete the old VPs will be automatically removed and you will only see one VP in the lists. vSAN VP will now publish all of the policies to the SPBM, Older hosts cannot support the new policies.\nVASA Troubleshooting through CLI-vCenter Server in vSAN 6.7 As we now understand that the VASA provider is being moved to the vCenter server , troubleshooting VASA provider over CLI is also on the vCenter server only and not on ESXi the vsanvpd.log now is moved into the health plugin directory of vcenter server .\nCheck if vsan-health service is running . root@photon-machine [ ~ ]# vmon-cli -s vsan-health Name: vsan-health Starttype: AUTOMATIC RunState: STARTED RunAsUser: vsan-health CurrentRunStateDuration(ms): 386462471 HealthState: HEALTHY FailStop: N/A MainProcessId: 11668\nTo Restart service : 1root@photon-machine [ ~ ]# **vmon-cli -r vsan-health** 2Completed Restart service request. Logs to check for issues with vsan VP 6.7 : 1**cd /var/log/vmware/vsan-health/** 2**/var/log/vmware/vsan-health ]# less vsanvp.log 3[ /var/log/vmware/vsan-health ]# less vmware-vsan-health-service.log | grep failed** ","link":"https://blogs.hareeshkg.bid/post/how-to-troubleshoot-issues-with-vsan-storage-policy-compliance-and-vasa-providers/","section":"post","tags":[],"title":"How to troubleshoot issues with vSAN  Storage policy compliance and VASA Providers ?"},{"body":"vSAN will declare the disks as permanently unavailable , dead or unmounted for multiple reasons . You will have further investigate on the logs to find out the case for such unmounts . Here is a small guide which should enable you to isolating the problem .\nIt is always good to start looking into the vmkernel.log and vobd.log for the drive (naa.ID) to see the reason for the Disk unmount or DG umounts . The vCenter ⇒ Cluster ⇒ Configure (Manage in 6.2) ⇒ Disk Management View should indicate the Disk or DG which is unhealthy failed or unmounted .\nOne of a case study from Frequent Disk and DG unmount in an ALL-Flash Deduplication and Compression enabled cluster Here is a case study for one of the clusters which kept unmounting DGs randomly . I will start with the log evidence for the cause of the Disk group unmount and learnt over weeks period that the cluster was using unsupported SAS-expanders with about 28 disks (including cache tier and capacity tier ) , however the controller in use can only handle 14 drives . Per : https://blogs.vmware.com/virtualblocks/2016/05/17/vmware-virtual-san-sas-expanders/ , SAS-Expanders are only supported with vSAN ready nodes which are pre-tested and validated for Saturation issues on SAS BUS , here in this case we used to see a pattern where the drives on random host part of the cluster started to report read failures and later the drive gets unmounted by vSAN after which the entire DG gets unmount because this is an All-Flash Dedup and compression enabled cluster and a huge resync is triggered . We assume that this is because the drives in the bus didnt get enough bandwidth to acknowledge the IOPS as other drives/DG on the host might have saturated the SAS-Bus .\n**Check vmkernel.log under /var/run/log :\nLook for Permanent Errors :** Disk Event permanent error for MD 52827e2f-7a2c-8931-e978-1e57a2ff9078 (naa.51402ec0104cc514:2) 2018-02-22T22:12:56.479Z cpu10:28470336)WARNING: LSOM: LSOMEventNotify:6861: Virtual SAN device 52827e2f-7a2c-8931-e978-1e57a2ff9078 is under permanent error. 2018-02-22T22:12:56.479Z cpu10:28470336)LSOM: LSOMLogDiskEvent:5602: Disk Event permanent error propagated for MD 52726770-6b21-aa35-3227-ba18bec45c89 (naa.51402ec0104cc517:2) 2018-02-22T22:12:56.479Z cpu10:28470336)WARNING: LSOM: LSOMEventNotify:6872: Virtual SAN device 52726770-6b21-aa35-3227-ba18bec45c89 is under propagated permanent error. 2018-02-22T22:12:56.479Z cpu10:28470336)LSOM: LSOMLogDiskEvent:5602: Disk Event permanent error propagated for MD 528605c7-70cd-b6df-78bc-1c6ea90f57d2 (naa.51402ec0104cc507:2) 2018-02-22T22:12:56.479Z cpu10:28470336)WARNING: LSOM: LSOMEventNotify:6872: Virtual SAN device 528605c7-70cd-b6df-78bc-1c6ea90f57d2 is under propagated permanent error. 2018-02-22T22:12:56.479Z cpu10:28470336)LSOM: LSOMLogDiskEvent:5602: Disk Event permanent error propagated for SSD 5256b4fa-5f83-5b4a-8e9c-ae49c87fa660 (naa.58ce38ee200a70d9:2) 2018-02-22T22:12:56.479Z cpu10:28470336)WARNING: LSOM: LSOMEventNotify:6872: Virtual SAN device 5256b4fa-5f83-5b4a-8e9c-ae49c87fa660 is under propagated permanent error. 2018-02-22T22:17:17.897Z cpu12:66491)WARNING: PLOG: PLOGPropagateErrorInt:2809: Ignored permanent error event on stashed 52827e2f-7a2c-8931-e978-1e57a2ff9078 state=0xc09\nYou may also see Power on Reset Errors :\n2018-02-22T22:28:02.296Z cpu37:66492)ScsiCore: 1705: Power-on Reset occurred on naa.51402ec0104cc514 2018-02-22T22:28:10.299Z cpu22:68625024)ScsiCore: 1705: Power-on Reset occurred on vmhba0:C2:T20:L0 2018-02-22T22:28:18.307Z cpu23:66492)ScsiCore: 1705: Power-on Reset occurred on naa.51402ec0104cc514 2018-02-22T22:28:26.310Z cpu24:68625024)ScsiCore: 1705: Power-on Reset occurred on vmhba0:C2:T20:L0 2018-02-22T22:28:34.318Z cpu30:66492)ScsiCore: 1705: Power-on Reset occurred on naa.51402ec0104cc514 2018-02-22T22:29:05.337Z cpu24:65910)ScsiCore: 1705: Power-on Reset occurred on vmhba0:C2:T20:L0 2018-02-22T22:29:28.355Z cpu8:66491)ScsiCore: 1705: Power-on Reset occurred on naa.51402ec0104cc514\nRead (0x28) or Write (0x2a) failure (SCSI errors) :\n2018-02-22T22:11:53.634Z cpu13:66491)ScsiDeviceIO: 2927: Cmd(0x439a1e94ba80) 0x28, CmdSN 0x4ae975ed from world 0 to dev \u0026quot;naa.51402ec0104cc514\u0026quot; failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x4 0x43 0x0. 2018-02-22T22:11:53.634Z cpu13:66491)NMP: nmp_ThrottleLogForDevice:3617: Cmd 0x28 (0x439a1e617180, 0) to dev \u0026quot;naa.51402ec0104cc514\u0026quot; on path \u0026quot;vmhba0:C2:T20:L0\u0026quot; Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL 2018-02-22T22:12:01.640Z cpu38:66492)ScsiDeviceIO: 2927: Cmd(0x43a20b6ead00) 0x28, CmdSN 0x4ae975f6 from world 0 to dev \u0026quot;naa.51402ec0104cc514\u0026quot; failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x4 0x43 0x0. 2018-02-22T22:12:01.640Z cpu38:66492)ScsiDeviceIO: 2927: Cmd(0x439d0083e640) 0x28, CmdSN 0x4ae975f5 from world 0 to dev \u0026quot;naa.51402ec0104cc514\u0026quot; failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x4 0x43 0x0. 2018-02-22T22:12:01.640Z cpu38:66492)NMP: nmp_ThrottleLogForDevice:3617: Cmd 0x28 (0x43a392885e40, 0) to dev \u0026quot;naa.51402ec0104cc514\u0026quot; on path \u0026quot;vmhba0:C2:T20:L0\u0026quot; Failed: H:0x5 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL 2018-02-22T22:12:09.646Z cpu35:66492)ScsiDeviceIO: 2927: Cmd(0x43a384c7e480) 0x28, CmdSN 0x4ae975fd from world 0 to dev \u0026quot;naa.51402ec0104cc514\u0026quot; failed H:0x5 D:0x0 P:0x0 Invalid sense data: 0x80 0x41 0x0. 2018-02-22T22:12:09.646Z cpu35:66492)NMP: nmp_ThrottleLogForDevice:3617: Cmd 0x28 (0x439d0083e640, 0) to dev \u0026quot;naa.51402ec0104cc514\u0026quot; on path \u0026quot;vmhba0:C2:T20:L0\u0026quot; Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0. Act:NONE 2018-02-22T22:12:09.646Z cpu35:66492)ScsiDeviceIO: 2927: Cmd(0x439d0083e640) 0x28, CmdSN 0x4ae975f5 from world 0 to dev \u0026quot;naa.51402ec0104cc514\u0026quot; failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x29 0x0.\nIn the Above example : We see that the the Disk “naa.51402ec0104cc514” started to report power on reset events , these events are usually seen whenever the disk stops responding to all queries / SCSI commands from the hosts , the hosts try to reset the Luns (naa.ID) to see if the disk can come back to a responding state , after retrying multiple times , vSAN marks the drive as PDL (permanent device loss) this may or may not be a genuine drive fault because of media errors or a full drive failure further investigation will also need to be done from H/W perspective . Here in this example it is very clear that the drive “naa.51402ec0104cc514” started to report read failures 0x28 continuously and host retried to reset the Lun to get it in accessible state , however later gives and marks it as permanently lost . Note* : If this an ALL-Flash vSAN with dedup and compression enabled cluster you are expected to see the entire diskgroup being un-mounted which is an expected behavior for a capacity tier SSD drive failure . In a non-dedup environment we expect to see only a failure on cache tier drive which causes DG unmount and any capcity teir drive failure should continue to function fine without the faulty disk . Here is a list of possible reason which can potentially mark Disks as failed The disk stopped responding to VSAN IO requests in a timely manner. When the reason for the disk failure declaration is due to IO timeout, the vmkernel log will report “maximum kernel-level retries exceeded” in association with the permanent error message. For example: LSOMCommon: IORETRYParentIODoneCB:1043: Throttled: split status Maximum kernel-level retries exceeded\nThe disk encountered a genuine fault such as a medium error. In these cases, the disk will be marked as faulted, even though the disk may still appear in the system. When encountering a medium error, Virtual SAN will mark the disk as being in a permanent error state. When a medium error is encountered, the problem will manifest in two ways: The vmkernel log will reflect that the permanent-error event as due to “I/O Error” WARNING: LSOMCommon: IORETRYParentIODoneCB:1466: Throttled: split status I/O error\nThe vmkernel log will report that at least one I/O operation to the storage device failed due to a medium error. This is reflected by sense-key 0x3: NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x2a (0x439fc298e0c0, 0) to dev “naa.5000cca072ac3b58” on path “vmhba2:C0:T14:L0” Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x15 0x0. Act:NONE\n- For more information on decoding SCSI error messages : [https://en.wikipedia.org/wiki/SCSI_command](https://en.wikipedia.org/wiki/SCSI_command) The disk was removed from the storage path. When this is encountered, the vmkernel log will report a PDL (permanent device loss) or APD (all paths down) condition associated with a device. The most common scenario is a disk going into PDL, and Virtual SAN will interpret this as a permanent condition and will mark the disk as permanently unavailable as IO will fail due to “not supported”\nWARNING: NMP: nmp_PathDetermineFailure:2961: Cmd (0x2a) PDL error (0x5/0x25/0x0) – path vmhba2:C2:T2:L0 device naa.600605b0099250d01da7c6a019312a53 – triggering path evaluation\nNMP: nmp_ThrottleLogForDevice:3286: Cmd 0x2a (0x439ee894cc00, 0) to dev “naa.600605b0099250d01da7c6a019312a53” on path “vmhba2:C2:T2:L0” Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0. Act:EVAL\nLSOMCommon: IORETRYCompleteIO:495: Throttled: 0x439ee7ea0c00 IO type 304 (WRITE) isOdered:NO since 20392 msec status Not supported\nWARNING: LSOM: LSOMEventNotify:6126: Virtual SAN device 52a89ef4-6c3f-d16b-5347-c5354470b465 is under permanent error.\nWhen the failure is due to APD instead of PDL (a comparatively rare scenario), the failure will be due to “status Not found“ For more information on APD and PDL behavior, please see : Permanent device loss and All-Paths-Down handling in vsphere 5.x and 6.x - The overall disk performance level deteriorated to the point that Virtual SAN marked the disks as offline to prevent system performance degradation. When this occurs, Virtual SAN will unmount the affected disks or disk groups. The vmkernel log will report that VSAN Device Monitor took the unmount action: VSAN Device Monitor: Unmounting VSAN diskgroup eui.2114aa100d00001\nPlease see Dying Disk handling in vSAN KB : [vSAN 6.1/5.5 Update 3 Disk Groups show as Unmounted in the vSphere Web Client (DDH) (2132079)](https://kb.vmware.com/s/article/2132079) ","link":"https://blogs.hareeshkg.bid/post/why-does-vsan-disks-get-unmounted/","section":"post","tags":[],"title":"Why does vSAN disks get unmounted ?"},{"body":"I have been asked about vSAN iSCSI target configuration and how the objects looks within vSAN ? Here is a small guide that gets you going with vSAN iSCSI configuration and how you get to see the actual luns within vSAN as vSAN objects .\nThe iSCSI lun creation and presentation is very simple , I have put together screenshots from my lab to create and export vSAN iSCSI luns to a Windows server 2012 box . Please follow the series of screenshots to create and setup iSCSI exports from vSAN to external physical machines . In the below example we do not see creation of initiator group , usually initiator groups are created to allow a set of Luns to a certain set of IQN to which the luns must be visible .\nNote: iSCSI Lun exports to virtual machines is unsupported officially , strictly refrain from presenting an iSCSI vSAN export to a VM in production environment .*\nLooking at the objects for the iSCSI Lun export thru vSAN. VIT (virtual iSCSI Target) is the service which runs on all the host is responsible to manage and facilitate the vSAN iSCSI target on all the hosts , I suggest you to see VMwareKB to troubleshoot “iSCSI target service”\nAfter creating first iSCSI target when we look at Target owner host we see below messages , this indicates that the object for the target iSCSI lun was created successfully , if there were any issues while creating the iSCSI target from vcenter server , please see the vmkernel.log on the owner host for more details\nVmkerne.log : “less /var/log/vmkernel.log | grep -i vit” : this the logs seen during the time of creating a target .\n2018-01-14T03:38:39.296Z cpu16:9176868 opID=a09e5a8d)World: 12235: VC opID VSAN-iSCSI-OP-52e79e76-7274-e36b-daa3-c5a03437e5b2 maps to vmkernel opID a09e5a8d 2018-01-14T03:38:44.440Z cpu17:9176879)Loading module vit ... 2018-01-14T03:38:44.441Z cpu17:9176879)Elf: 2043: module vit has license VMware 2018-01-14T03:38:44.450Z cpu17:9176879)VIT: Resetting config options 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:497: Size of global heap 0x4307450f5000 : 8681472 bytes 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:498: Size of target heap: 952320 bytes 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:499: Size of LUN heap: 4601856 bytes 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:500: Size of ctl_softc 12256 bytes. 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:501: Size of cfiscsi_target 928 bytes. 2018-01-14T03:38:44.450Z cpu17:9176879)vit: HeapLogSizes:502: Size of ctl_lun 118584 bytes. 2018-01-14T03:38:44.453Z cpu17:9176879)vit: VitFssBeInit:543: Number of threads for IO task queue: 32 2018-01-14T03:38:44.457Z cpu17:9176879)Mod: 4968: Initialization of vit succeeded with module ID 119. 2018-01-14T03:38:44.457Z cpu17:9176879)vit loaded successfully. 2018-01-14T03:40:36.410Z cpu18:9177736 opID=40c602bf)World: 12235: VC opID VSAN-iSCSI-OP-52b07c6f-192f-031e-1c15-011af0ab1edf maps to vmkernel opID 40c602bf 2018-01-14T03:40:38.580Z cpu1:9177019)vit: cfiscsi_target_find_or_create:2542: Created heap RDP-win for target RDP-win. Heap pointer: 0x4305e6412000. 2018-01-14T03:40:42.078Z cpu22:9177018)vit: VitFSSBEIoctl_Create:2207: LUN heap cfss0 created (heap ID: 0x430750e13000). 2018-01-14T03:40:42.078Z cpu22:9177018)vit: VitFSSBE_GetInfo:1927: GetInfo size 10737418240 bs=512 size_blocks 20971520 2018-01-14T03:42:53.006Z cpu13:68816)vit: VITVsiGetIoStats:14820: Throttled: Perf stats isn't initialized for ctl LUN 0 2018-01-14T03:42:53.006Z cpu13:68816)vit: VITVsiGetBwStats:14711: Throttled: Perf stats isn't initialized for ctl LUN 0 2018-01-14T03:42:53.006Z cpu13:68816)vit: VITVsiGetLatencyStats:14939: Throttled: Perf stats isn't initialized for ctl LUN 0\nUsing certain esxcli vsan commands to get more information about the iSCSI targets 1***localcli vsan iscsi status get 2****Status: Enabled: true**** 3 4localcli vsan iscsi target list*** 5Alias iSCSI Qualified Name (IQN) Interface Port Authentication type LUNs Is Compliant UUID I/O Owner UUID 6------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7**\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;RDP-win\u0026lt;/span\u0026gt;**\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;*iqn.1998-01.com.vmware:22455fef-6b44-e32b-eb2a-f5535af9a5de*\u0026lt;/span\u0026gt; vmk1 3260 No-Authentication 1 true **\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;34d15a5a-7ff7-924b-776c-ecf4bbec91a8\u0026lt;/span\u0026gt;** 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 8 9**\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;Win-iSCSI-Lun2\u0026lt;/span\u0026gt;***\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;iqn.1998-01.com.vmware:0f4751c1-2a44-bb90-83e4-f567cfc711d4\u0026lt;/span\u0026gt;* vmk1 3260 No-Authentication 1 true **\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;62d65a5a-48dd-477d-8c86-ecf4bbec65d8\u0026lt;/span\u0026gt;** 5938de9a-e35b-d745-c9ff-ecf4bbec65d8 10 11***localcli vsan iscsi homeobject get 12*** 13Namespace Information: 14 UUID:**\u0026lt;span style=\u0026#34;color: #008000;\u0026#34;\u0026gt; bfd05a5a-0dc2-53da-444a-ecf4bbec91a8\u0026lt;/span\u0026gt;** 15 Policy: ((\u0026#34;stripeWidth\u0026#34; i1) (\u0026#34;cacheReservation\u0026#34; i0) (\u0026#34;proportionalCapacity\u0026#34; (i0 i100)) (\u0026#34;hostFailuresToTolerate\u0026#34; i1) (\u0026#34;forceProvisioning\u0026#34; i0) (\u0026#34;spbmProfileId\u0026#34; \u0026#34;aa6d5a82-1c88-45da-85d3-3d74b91a5bad\u0026#34;) (\u0026#34;spbmProfileGenerationNumber\u0026#34; l+2) (\u0026#34;CSN\u0026#34; l1) (\u0026#34;spbmProfileName\u0026#34; \u0026#34;vSAN Default Storage Policy\u0026#34;)) 16 Is Compliant: true 17 18***localcli vsan iscsi target get --alias*****\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;RDP-win\u0026lt;/span\u0026gt;** 19Target Information: 20 IQN: iqn.1998-01.com.vmware:22455fef-6b44-e32b-eb2a-f5535af9a5de 21 Alias: RDP-win 22 UUID: **\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;34d15a5a-7ff7-924b-776c-ecf4bbec91a8\u0026lt;/span\u0026gt;** 23 Is Compliant: true 24 Policy: ((\u0026#34;stripeWidth\u0026#34; i1) (\u0026#34;cacheReservation\u0026#34; i0) (\u0026#34;proportionalCapacity\u0026#34; (i0 i100)) (\u0026#34;hostFailuresToTolerate\u0026#34; i1) (\u0026#34;forceProvisioning\u0026#34; i0) (\u0026#34;spbmProfileId\u0026#34; \u0026#34;aa6d5a82-1c88-45da-85d3-3d74b91a5bad\u0026#34;) (\u0026#34;spbmProfileGenerationNumber\u0026#34; l+2) (\u0026#34;CSN\u0026#34; l1) (\u0026#34;spbmProfileName\u0026#34; \u0026#34;vSAN Default Storage Policy\u0026#34;)) 25 Interface: vmk1 26 Port: 3260 27 Authentication type: No-Authentication 28 Authentication userid: 29 Authentication mutual-userid: 30 LUNs: 1 31 I/O owner UUID: 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 32 I/O owner: IP-Address of the owning host . 33 Allowed initiators: iqn.1991-05.com.microsoft:rdp 34 Allowed initiator groups: vit.conf :: cat /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/etc/vit.conf 1generation 8 2auth-group default { auth-type none} 3auth-group 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 { auth-type none} 4auth-group 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 { auth-type none} 5portal-group default { discovery-auth-group no-authentication listen vmk1:3260} 6portal-group pg-vmk1-3260 { discovery-auth-group no-authentication listen vmk1:3260} 7target **\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;iqn.1998-01.com.vmware:22455fef-6b44-e32b-eb2a-f5535af9a5de\u0026lt;/span\u0026gt;** 8{ alias \u0026#34;***\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;RDP-win\u0026lt;/span\u0026gt;***\u0026#34; portal-group pg-vmk1-3260 9 auth-group 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 10 option uuid 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 11 option owner-id 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 12 lun 1 { backend vmdk path 34d15a5a-7ff7-924b-776c-ecf4bbec91a8/\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;***39d15a5a-c635-9e1b-0c92-ecf4bbec91a8.vmdk***\u0026lt;/span\u0026gt; 13 size **20971520** 14 option lun-alias \u0026#34;\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;***RDP-win***\u0026lt;/span\u0026gt;\u0026#34; 15 } 16} 17target **\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;iqn.1998-01.com.vmware:0f4751c1-2a44-bb90-83e4-f567cfc711d4\u0026lt;/span\u0026gt;** { 18 alias \u0026#34;Win-iSCSI-Lun2\u0026#34; 19 portal-group pg-vmk1-3260 20 auth-group 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 21 option uuid 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 22 option owner-id 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 23 lun 3 { backend vmdk path 62d65a5a-48dd-477d-8c86-ecf4bbec65d8/**\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;*67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8.vmdk*\u0026lt;/span\u0026gt;** 24 size **20971520** 25 option lun-alias \u0026#34;\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;***Win-iSCSI-Lun2***\u0026lt;/span\u0026gt;\u0026#34; 26 } 27} **Looking at the Actual objects for the Luns which are created as objects in vSAN ** change directory to : /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/targets : /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/bfd05a5a-0dc2-53da-444a-ecf4bbec91a8/targets] ls -l total 16 lrwxrwxrwx 1 root root 42 Jan 14 03:40 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 -\u0026gt; ../../34d15a5a-7ff7-924b-776c-ecf4bbec91a8 lrwxrwxrwx 1 root root 42 Jan 14 04:02 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 -\u0026gt; ../../62d65a5a-48dd-477d-8c86-ecf4bbec65d8\n1cd to **\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;34d15a5a-7ff7-924b-776c-ecf4bbec91a8\u0026lt;/span\u0026gt;** (Absolute Path : */vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/34d15a5a-7ff7-924b-776c-ecf4bbec91a8*) 2 3\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/34d15a5a-7ff7-924b-776c-ecf4bbec91a8\u0026lt;/span\u0026gt;] ls -l 4total 0 5-rw------- 1 root root 521 Jan 14 03:40 **\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;39d15a5a-c635-9e1b-0c92-ecf4bbec91a8.vmdk 6\u0026lt;/span\u0026gt;**cat \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;**39d15a5a-c635-9e1b-0c92-ecf4bbec91a8.vmdk**\u0026lt;/span\u0026gt; | grep vsan 7RW 20971520 VMFS \u0026#34;vsan://\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;**39d15a5a-c635-9e1b-0c92-ecf4bbec91a8**\u0026lt;/span\u0026gt;\u0026#34; \u0026lt;--- This is the destination object where the data is stored for the Lun \u0026#34;***\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;RDP-win\u0026lt;/span\u0026gt;***\u0026#34; . 8 9[root@is-tse-d156:/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/bfd05a5a-0dc2-53da-444a-ecf4bbec91a8/targets] ls -l 10total 16 11lrwxrwxrwx 1 root root 42 Jan 14 03:40 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 -\u0026gt; ../../34d15a5a-7ff7-924b-776c-ecf4bbec91a8 12lrwxrwxrwx 1 root root 42 Jan 14 04:02 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 -\u0026gt; ../../62d65a5a-48dd-477d-8c86-ecf4bbec65d8 13 14cd : \u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;**62d65a5a-48dd-477d-8c86-ecf4bbec65d8**\u0026lt;/span\u0026gt; (Absolute Path : */vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/62d65a5a-48dd-477d-8c86-ecf4bbec65d8*) 15 16\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;/vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/62d65a5a-48dd-477d-8c86-ecf4bbec65d8\u0026lt;/span\u0026gt;] ls -l 17total 0 18-rw------- 1 root root 521 Jan 14 04:02 **\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8\u0026lt;/span\u0026gt;\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;.vmdk\u0026lt;/span\u0026gt;** 19 20cat ***\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8.vmdk\u0026lt;/span\u0026gt;*** | grep vsan 21RW 20971520 VMFS \u0026#34;vsan://***\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8\u0026lt;/span\u0026gt;***\u0026#34; \u0026lt;--- This is the destination object where the data is stored for the Lun \u0026#34;***\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;Win-iSCSI-Lun2\u0026lt;/span\u0026gt;***\u0026#34; **Using esxcli vsan debug objects list to dump all object attributes to a file , we can look at the placement of the components for each Lun objects and component health .** esxcli vsan debug object list \u0026gt; /tmp/list_all_object.txt less the output file “less /tmp/list_all_object.txt” , search for “39d15a5a-c635-9e1b-0c92-ecf4bbec91a8“ Object UUID: 39d15a5a-c635-9e1b-0c92-ecf4bbec91a8 Version: 5 Health: healthy Owner: X.X.X.X (Host IP/FQDN) Policy: spbmProfileName: vSAN Default Storage Policy forceProvisioning: 0 spbmProfileGenerationNumber: 2 stripeWidth: 1 cacheReservation: 0 CSN: 2 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad proportionalCapacity: 0 hostFailuresToTolerate: 1\nConfiguration:\nRAID_1 Component: 39d15a5a-1794-5d1c-c279-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 52f3a35d-2937-9811-81b0-026815670d32, Disk Name: naa.5002538c4044d87f:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d157.isl.vmware.com\nComponent: 39d15a5a-afc7-5e1c-d3d4-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 10737418240 (10.00GB), Disk UUID: 52bf76f2-4863-2447-b919-1d7923a822e3, Disk Name: naa.5002538c4044d6ad:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d155.isl.vmware.com\nWitness: 39d15a5a-048a-5f1c-5381-ecf4bbec91a8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 525b1f66-2d22-7897-d936-6eefd92c7019, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d156.isl.vmware.com\nType: vdisk Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/targets/34d15a5a-7ff7-924b-776c-ecf4bbec91a8/52dc6cdc-d19d-ac5c-1b32-c5c94383310c.vmdk (Missing) Group UUID: 34d15a5a-7ff7-924b-776c-ecf4bbec91a8 Directory Name: (null)\nless the previous output file “less /tmp/list_all_object.txt” , search for “62d65a5a-48dd-477d-8c86-ecf4bbec65d8“ 1Object UUID: 67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8 2 Version: 5 3**Health: healthy** 4 Owner: X.X.X.X (Host IP/FQDN) 5 Policy: 6** spbmProfileName: vSAN Default Storage Policy** 7 forceProvisioning: 0 8 spbmProfileGenerationNumber: 2 9 stripeWidth: 1 10 cacheReservation: 0 11 CSN: 4 12 spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad 13 proportionalCapacity: 0 14 hostFailuresToTolerate: 1 15 16Configuration: 17 18**RAID_1** 19 Component: 67d65a5a-a21d-5c42-d279-ecf4bbec91a8 20**Component State: ACTIVE**, Address Space(B): 10737418240 (10.00GB), Disk UUID: 5270b13f-a6a0-50cf-0cb3-88d86b7d323e, Disk Name: naa.5002538c4044d6ab:2 21 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d155.isl.vmware.com 22 23 Component: 67d65a5a-9967-5d42-3ad0-ecf4bbec91a8 24**Component State: ACTIVE**, Address Space(B): 10737418240 (10.00GB), Disk UUID: 525b1f66-2d22-7897-d936-6eefd92c7019, Disk Name: naa.5002538c4044d6a3:2 25 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d156.isl.vmware.com 26 27 Witness: 67d65a5a-7227-5e42-31ab-ecf4bbec91a8 28**Component State: ACTIVE**, Address Space(B): 0 (0.00GB), Disk UUID: 529e9a5d-a5d8-6a18-3933-ed69eca58c36, Disk Name: naa.5002538c4044d6ae:2 29 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 4194304 (0.00GB), Host Name: is-tse-d157.isl.vmware.com 30 31**Type: vdisk** 32 Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/.iSCSI-CONFIG/targets/62d65a5a-48dd-477d-8c86-ecf4bbec65d8/5239144d-34bc-439c-1907-2c541a56a141.vmdk (Missing) 33 Group UUID: 62d65a5a-48dd-477d-8c86-ecf4bbec65d8 34 Directory Name: (null) RVC command to look at the objects from vCenter server RVC session ***Note*: All iSCSI target will be listed under unassociated objects , be extra careful while deleting associated objects as you may accidentally delete vSAN iSCSI home folder and the target VMDKs and they cannot be recovered once deleted .*** 1/localhost/vsan66-AF-DELL/computers/vsan66-ALL-FLASH-DELL\u0026gt; vsan.obj_status_report . -t 2 3Unassociated objects 41a39c759-70d6-ea0a-29f4-ecf4bbec65d8 5d9e59b59-9031-180b-6000-ecf4bbec65d8 679ca385a-f785-a610-5038-ecf4bbec65d8 7**\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;39d15a5a-c635-9e1b-0c92-ecf4bbec91a8\u0026lt;/span\u0026gt;** \u0026lt;---------------iSCSI target Lun 1 object for **\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;RDP-win\u0026lt;/span\u0026gt;** vmdk 8**\u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;67d65a5a-b1e9-fc41-a4d2-ecf4bbec91a8\u0026lt;/span\u0026gt;** \u0026lt;---------------iSCSI target Lun 3 object \u0026lt;span style=\u0026#34;color: #993300;\u0026#34;\u0026gt;**Win-iSCSI-Lun2**\u0026lt;/span\u0026gt; vmdk 9\u0026lt;span style=\u0026#34;color: #339966;\u0026#34;\u0026gt;**bfd05a5a-0dc2-53da-444a-ecf4bbec91a8**\u0026lt;/span\u0026gt; \u0026lt;---------------iSCSI home folder object ","link":"https://blogs.hareeshkg.bid/post/vsan-iscsi-targets-configuration-deep-dive/","section":"post","tags":[],"title":"vSAN iSCSI targets configuration deep dive"},{"body":"After you setup a new vSAN/VxRAIL/VxRACK HCI deployment , you will definitely need to benchmark your cluster before moving it to production . Here is your small guide to “vSAN Performance Benchmark” using a famous useful tool called HCI Bench which is freely available on VMware Labs website .\nYou may be wondering what is the recommended and correct way to test the performance and review performance on a hyper-converged infrastructure .\nIf you have not gone thru my previous posts about understanding-vm-storage-policies and understanding-vsan-component-placement-sizing I highly encourage you to go thru that topic which will give you a deep insight on how components are created , placed and how we can view them .\nHow NOT to assess Performance Now lets discuss how we can run performance benchmark to validate a vSAN cluster which was newly deployed and about to be staged for your production .Many people prefer to deploy a Windows or Linux VM running tools available at the guest level like IOmeter to do performance benchmark which is okay in one way to asses your performance , however we are NOT going to assess the cluster properly.We all understand that the vSAN is a hyper-converged infrastructure , the data is placement is purely controlled and driven by SPBM (Storage policy based management ).\nExample : Lets say we have a new 8 node cluster and lets also consider your test benchmark VM (Windows or Linux which is going to probably run some performance bench marking tool) was deployed with the “vSAN Default Storage Policy” and imagine your vmdk size where you will be running the performance test is around 50GB , we should probably be seeing 3 components (2 Data components and a witness) which is going to reside only three hosts and 3 physical disks on those three hosts . When you start running your performance tests , all the workload and IO are only going to hit the two data components on those two hosts where the data components are placed and never going to hit any other hosts nor their disk . If the tests are super heavy workloads , we are only going to beat down those two disks which contain your data components and chances are pretty high that you will start seeing “Component congestion” and “SSD Congestion” which is artificially induced as all IOPS are specifically targeted on those two components residing those two disks .\nConclusion : All your IOPS were targeting only two individual capacity tier disks residing on a disk group belonging to two hosts in the entire cluster where the two component resides and never touched any disks on your other 6 nodes (as per above example of a 8 node cluster)\nA very high workload can cause Component congestion against the components for the VM and not truly testing overall vSAN performance . The results will not be accurate and may report poor performance because of the induced component congestion (Due to disks being highly busy and not able to cope up with new incoming IO) and also SSD congestion (If there were too many outstanding IO and SSD unable to Destage the data from the cache tier to capacity tier disks since the capacity disks are still busy serving existing IOPS for the components in question)\nAlso considering this being a 8 node cluster which multiple Disk Groups , we never really hit any other node over the vSAN network and their disk groups at all to see if everyone is healthy .\nHCI Bench This is one of the best I have seen which can run a good benchmark for a vSAN cluster , this tool can deploy multiple VMs on all the hosts with each VM having multiple vDisks and components will be spread across all the disks /disk groups on all member nodes , this ensures that we push traffic thru vSAN network interfaces across all nodes , generate workloads which targets all the SSD cache tier and SSD/MD capacity (All-Flash or Hybrid ) . Here is how you initially deploy the OVA , configure it to run the BenchMark tool , we will later see how we understand the Results post running the test in the next section which I will be adding soon .\nAccept Technical Preview License and download it from https://labs.vmware.com/flings/hcibench\nGood to have Instructions handy which may be needed later Start to import the HCI Bench OVA Name your Management HCI bench tool VM Choose your cluster or resource pool to which this HCI management VM will be deployed upon Accept and hit next for the additional settings for this OVA Accept EULA Datastore where this HCI management bench tool is going to be deployed , this is not necessary be the destination datastore where the performance bench tests will be run . Here it is very Important to understand what is Private and Public network and this will vary depending on the environment where this is being deployed . Please understand that public network is the network where the HCI management VM is deployed through which you can run and manage your Benchmark from , ensure you have a machine in the same network from which this bench tool can be managed from . The private network is where the VDBench machines will be deployed for performance benchmark and these needs to be able to communicate with the HCI management machine thru this second nick , also it is good to have a DHCP server on this private network or HCI bench tool has the power to assign private IPs thru this network , this can be configured later during initial configuration.\nReview your OVA deployment settings Power on your HCI Bench VM wait for it to show up the IP address post boot up Open your HCI Bench Over the Browser using its Port and port 8080 Once you enter into the configuration page (HCI Bench Management IP address : 8080) , you will need to fill in your vcenter hostname IP / username/password to authenticate to VC , also fill in details of the datacenter , customer and network where your test VDBench machines will be deployed to run the performance bench tests . You may choose deploy the VMs directly on the hosts if you were not using VDS (DV switches) in the environment , also make sure your DRS is not automatic if you were going to deploy directly on the hosts , this can be left unchecked . Click on download button the HCI bench tool and it should direct you to the Oracle website where you can download this Accept the EULA and download the VDBench zip file from the Oracle website Finish uploading the VDbenchp zip file to the HCL bench management VM Validate and kick of the bench mark test Run Post Deployment and Test completion , you must be seeing a bunch of vdbench test machines with multiple disks attached to them on which the benchmark tests were run , the below test was run with easyMode on the VDbench test which deployed about 6 VMs on the three node vSAN cluster and each VM had 9 disks on which the benchmark test was run .\nThis blog only covers steps to download , deploy and run a easyrun on HCI-Bench , however doesnot explain you how to run different tests and achieve best performance out of the cluster . I highly recommend you to read the Blog : How to run HCIBench to Achieve Best performance on vSAN ? for better understanding on HCI-Bench advanced test-run.\n","link":"https://blogs.hareeshkg.bid/post/why-and-how-to-deploy-vsan-hci-bench-for-vsan/","section":"post","tags":[],"title":"Why and How to Deploy vSAN HCI Bench for vSAN?"},{"body":"I highly encourage everyone to go thru the Blog/Article about “understanding-vsan-objects-and-component” which explains how components are created based on the policy that you create . So that you understand the fundamentals of Objects and components which will help you “Understanding VM-Storage-Policies” .\nLets pick up a single virtual machines try and apply different policy , look at the object placement view and number of components created their placement , object reservation and number of child components created . Look at this from Webclient / RVC and from ESXi perspective and understand the data/outputs . We really need to know the RVC method and ESXCLI method because during an outage we might have lost vcenter , to look at status of the objects esxcli commands will be still helpful .\nView Component placement using vsphere Webclient vsphere webclient view is mostly user friendly to view the object placement for all the VMs and the health status of each object . We can do so by navigating to the **vSAN-cluster ⇒ Monitor ⇒ vSAN ⇒ Virtual Objects **or the other way is by looking at individual VMs and their Object Placement by navigating thru vSAN-cluster ⇒ VM ⇒ Monitor ⇒ Policies .\nView Component Placement using RVC Component placement for virtual machines can be easily retrieved using RVC command “vsan.vm_object_info” , first login to RVC session for vCenter server (vCSA/windows VC) see :Link\nNow once you are in RVC , change Directory to VM directory from root location : cd “/localhost/Datacenter-NAME”/vms” and then do an ls to see list of vms . You will see that every VM is associated with a number prefixed to it , its easy for you to run this command against the prefixed number To view vm object placement information for each VM simply run {vsan.vm_object_info “Pre-fix-number” } see below screenshots.\nView Component Placement Using ESXCLI (Version 6.5 and above) Component placement for all objects on vSAN can be exported to a file and then later viewed offline using a command on any of the hosts part of the vSAN cluster in question :\n“**esxcli vsan debug object list ** **\u0026gt; **/tmp/objects.txt” , this helps us export all output from command “esxcli vsan debug object list” to file called “objects.txt” under “/tmp/” directory . This can be viewed later by either exporting it to local system using winscp or can be viewed in the ESXi host using cat or less command . Below is a sample output .\nesxcli vsan debug object list | less\nObject UUID: f205b559-3185-a000-db32-ecf4bbec65d8 Version: 5 Health: healthy Owner: is-tse-d155.isl.vmware.com Policy: cacheReservation: 0 stripeWidth: 1 spbmProfileGenerationNumber: 2 forceProvisioning: 0 spbmProfileName: vSAN Default Storage Policy hostFailuresToTolerate: 1 proportionalCapacity: [0, 100] spbmProfileId: aa6d5a82-1c88-45da-85d3-3d74b91a5bad CSN: 99 SCSN: 97\nConfiguration:\nRAID_1 Component: f205b559-3e35-1901-59e7-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 273804165120 (255.00GB), Disk UUID: 529e9a5d-a5d8-6a18-3933-ed69eca58c36, Disk Name: naa.5002538c4044d6ae:2 Votes: 1, Capacity Used(B): 452984832 (0.42GB), Physical Capacity Used(B): 444596224 (0.41GB), Host Name: is-tse-d157.isl.vmware.com Component: f205b559-1c8f-1a01-1fb3-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 273804165120 (255.00GB), Disk UUID: 5270b13f-a6a0-50cf-0cb3-88d86b7d323e, Disk Name: naa.5002538c4044d6ab:2 Votes: 1, Capacity Used(B): 448790528 (0.42GB), Physical Capacity Used(B): 440401920 (0.41GB), Host Name: is-tse-d155.isl.vmware.com Witness: f205b559-f362-1b01-d0e5-ecf4bbec65d8 Component State: ACTIVE, Address Space(B): 0 (0.00GB), Disk UUID: 525b1f66-2d22-7897-d936-6eefd92c7019, Disk Name: naa.5002538c4044d6a3:2 Votes: 1, Capacity Used(B): 12582912 (0.01GB), Physical Capacity Used(B): 8388608 (0.01GB), Host Name: is-tse-d156.isl.vmware.com\nType: vmnamespace Path: /vmfs/volumes/vsan:523d5e5605a4d751-0c3304ae7a42599b/ Group UUID: 00000000-0000-0000-0000-000000000000 Directory Name: ESXi-66-Stretch-7\nView Objects thru ESXCLI (version 6.0 and below) The “**esxcli vsan debug object list” **works only on hosts running ESXi 6.5 and above this is unavailable on the 6.0/5.5 releases . However the object placement can be still pulled with another command which can be run as a python script : “**python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc \u003e /tmp/Object-health.txt**” , we later use less command to look at individual objects by searching with the Object for either vDisk UUID or VM-NameSpace UUID . EX: python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc \u0026gt; /tmp/health.txt\n1DOM Object reference: 2===================== 3Object 5f8fb459-7807-2d00-67d3-0cc47ac2b550 (v4, owner: esxi03, policy: {\u0026#39;spbmProfileGenerationNumber\u0026#39;: 4, \u0026#39;forceProvisioning\u0026#39;: 0, \u0026#39;cacheReservation\u0026#39;: 0, \u0026#39;checksumDisabled\u0026#39;: 0, \u0026#39;hostFailuresToTolerate\u0026#39;: 1, \u0026#39;stripeWidth\u0026#39;: 1, \u0026#39;spbmProfileId\u0026#39;: \u0026#39;aa6d5a82-1c88-45da-85d3-3d74b91a5bad\u0026#39;, \u0026#39;proportionalCapacity\u0026#39;: 0}): 4 Configuration 5 RAID_1 6 Component: 5f8fb459-c99f-a500-afd9-0cc47ac2b550 (state = 5, addr space = 16106127360 (15.00GB), disk = 5217932d-cbac-3ba3-74ec-63f5b845b6f2, 7 votes = 1, used = 29360128 (0.00GB), physUsed = 29360128 (0.00GB), hostname = esxi03) 8 Component: 5f8fb459-1f9c-a700-a485-0cc47ac2b550 (state = 5, addr space = 16106127360 (15.00GB), disk = 522babe3-1d47-5801-06bb-e24e9d8ee174, 9 votes = 1, used = 29360128 (0.00GB), physUsed = 29360128 (0.00GB), hostname = esxi01) 10 Witness: 5f8fb459-2df8-a800-8209-0cc47ac2b550 (state = 5, addr space = 0 (0.00GB), disk = 52797061-4f37-8319-4713-63d073b5fbb4, 11 votes = 1, used = 4194304 (0.00GB), physUsed = 4194304 (0.00GB), hostname = esxi02) Lets now see what we understand the differences between all different policy by applying to the same VM and see how the components are created /placed and some key information on the output from RVC and ESXCLI commands that we run. We have choosen a VM called “Windows_10” Lets see the comparison between different Polices.\nVirtual SAN Default Storage Policy [ ](/images/understanding-vm-storage-policies-vsan-default-pol.png) RVC : In this example we see :\nForce-provisioning= 0 , hence any VM that is created with this policy will need to strictly adhere to the policy rules and will not be forcefully created if we do not have enough number of hosts/diskgroups/disks to create and place the object . Object Creation is expected to fail .\n**CacheReservation = 0 **, hence all VM created with this policy will not have any reservation for reads in the available read cache on your Cache Tier drive in the Disk group .\n**ChecksumDisabled = 0 **, All VMs created with this policy will use the embedded checksum with vSAN , hence all reads against the objects using this policy will be validated against the checksum\nHostFailuresToTolerate = 1 ,this is number of hosts/disks/disk group failure that the object can take before it goes inaccessible for use .\n**stripeWidth = 1 , **This tells that with how many disks are we going to stripe the data across within a DG .\n**spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad , **This is the unique UUID assigned to this Policy within vSAN.\n**proportionalCapacity = 0 , **(see vdisk) this tell us that the Object reservation for any VM created with this policy is 0 , hence the objects are fully thin and will grow as and when the data fills within the objects .\nNote**: In vSAN creating of virtual disks with thin / thick is not recommended for space reservation , this needs to be defined thru Object Space reservation .\nSome Important Observations from Below example :\nWe see that there is a slight difference the way this object placement is interpreted on the RVC output and ESXCLI output . We see that the physical location where the component is placed is shown with the “naa.xxxxxxxxx” the LUN ID of the physical disk vs the disk UUID on the ESXCLI output , also on the RVC we see that SSD (cache tier) which is backing this Magnetic disk’s DiskGroup .\nWhen we look at the vDISK output from RVC and compare it with ESXCLI output , we see an additional section on the ESXCLI output EX : “used = 47571795968 (44.00GB), physUsed = 47571795968 (44.00GB)” , This tells us that the reserved and physical disk utilization by this component is same . You will see the difference when have object space reservation . The used space will be the actual size of the VMDK allocated space during the time of vDISK creation when the Object space reservation is 100% .\nVM Windows_10:\nNamespace directory :\nDOM Object: 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452 (v3, owner: 10.109.9.102, policy: spbmProfileGenerationNumber = 4, forceProvisioning = 0, cacheReservation = 0, checksumDisabled = 0, hostFailuresToTolerate = 1, stripeWidth = 1, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, proportionalCapacity = [0, 100]) RAID_1 Component: b398a759-303c-a6e3-c09d-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.103, md: naa.55cd2e404c211107, ssd: naa.55cd2e404c20d56d, votes: 1, usage: 0.4 GB) Component: 609eb859-c975-1b18-6be1-0cc47ac2b546 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c212a4c, ssd: naa.55cd2e404c20d56f, votes: 1, usage: 0.4 GB) Witness: 629eb859-8d15-b76e-c30b-0cc47ac2b546 (state: ACTIVE (5), host: 10.109.9.102, md: naa.55cd2e404c210ffd, ssd: naa.55cd2e404c20ce55, votes: 1, usage: 0.0 GB)\nvDISK :\nDisk backing: [vsanDatastore] 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452/Windows_10.vmdk DOM Object: 1daa5659-ac55-70f6-bc12-0cc47ac2b452 (v3, owner: 10.109.9.102, policy: spbmProfileGenerationNumber = 4, forceProvisioning = 0, cacheReservation = 0, checksumDisabled = 0, hostFailuresToTolerate = 1, stripeWidth = 1, spbmProfileId = aa6d5a82-1c88-45da-85d3-3d74b91a5bad, proportionalCapacity = 0) RAID_1 Component: b898a759-f8d6-0092-b302-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.103, md: naa.55cd2e404c212d55, ssd: naa.55cd2e404c20d56d, votes: 1, usage: 44.3 GB) Component: 7d00ae59-ffba-5b7d-c6cd-0cc47ac2b550 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c211ff5, ssd: naa.55cd2e404c20d56f, votes: 1, usage: 44.3 GB) Witness: e89eb859-8dac-b7c1-4081-0cc47ac2b550 (state: ACTIVE (5), host: 10.109.9.102, md: naa.55cd2e404c210ffd, ssd: naa.55cd2e404c20ce55, votes: 1, usage: 0.0 GB)\nESXi output from “python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc” 1Namespace directory : 2 3Object 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452 (v4, owner: esxi02, policy: {\u0026#39;spbmProfileGenerationNumber\u0026#39;: 4, \u0026#39;forceProvisioning\u0026#39;: 0, 4\u0026#39;cacheReservation\u0026#39;: 0, \u0026#39;checksumDisabled\u0026#39;: 0, \u0026#39;hostFailuresToTolerate\u0026#39;: 1, \u0026#39;stripeWidth\u0026#39;: 1, 5\u0026#39;spbmProfileId\u0026#39;: \u0026#39;aa6d5a82-1c88-45da-85d3-3d74b91a5bad\u0026#39;, \u0026#39;proportionalCapacity\u0026#39;: [0, 100]}): 6 Configuration 7 RAID_1 8 Component: b398a759-303c-a6e3-c09d-0cc47ac2b452 (state = 5, addr space = 273804165120 (255.00GB), \u0026lt;span style=\u0026#34;color: #000000;\u0026#34;\u0026gt;disk = 52ca462c-7d81-3e39-7bcc-858fe854fb81\u0026lt;/span\u0026gt;, 9 votes = 1, used = 461373440 (0.00GB), physUsed = 461373440 (0.00GB), hostname = esxi03) 10 Component: 609eb859-c975-1b18-6be1-0cc47ac2b546 (state = 5, addr space = 273804165120 (255.00GB), \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;\u0026lt;span style=\u0026#34;color: #000000;\u0026#34;\u0026gt;disk = 52f67161-bba6-9e7c-2a98-79bd160d14e\u0026lt;/span\u0026gt;6\u0026lt;/span\u0026gt;, 11 votes = 1, used = 452984832 (0.00GB), physUsed = 452984832 (0.00GB), hostname = esxi01) 12 Witness: 629eb859-8d15-b76e-c30b-0cc47ac2b546 (state = 5, addr space = 0 (0.00GB), disk = 52b7c9b9-01c1-1770-22af-7f9b78ee3d2f, 13 votes = 1, used = 4194304 (0.00GB), physUsed = 4194304 (0.00GB), hostname = esxi02) 14 15vDISK : 16 17Object 1daa5659-ac55-70f6-bc12-0cc47ac2b452 (v4, owner: esxi02, policy: {\u0026#39;spbmProfileGenerationNumber\u0026#39;: 4, \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;\u0026#39;forceProvisioning\u0026#39;: 0, 18\u0026#39;cacheReservation\u0026#39;: 0, \u0026#39;checksumDisabled\u0026#39;: 0, \u0026#39;hostFailuresToTolerate\u0026#39;: 1, \u0026#39;stripeWidth\u0026#39;: 1, 19 \u0026#39;spbmProfileId\u0026#39;: \u0026#39;aa6d5a82-1c88-45da-85d3-3d74b91a5bad\u0026lt;/span\u0026gt;\u0026#39;, \u0026#39;\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;proportionalCapacity\u0026#39;: 0\u0026lt;/span\u0026gt;}): 20 Configuration 21 RAID_1 22 Component: b898a759-f8d6-0092-b302-0cc47ac2b452 (state = 5, addr space = 214748364800 (200.00GB), \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;disk = 5280ff9a-9880-8549-ad8a-5ee212ee21ee\u0026lt;/span\u0026gt;, 23 votes = 1,\u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt; used = 47571795968 (44.00GB), physUsed = 47571795968 (44.00GB)\u0026lt;/span\u0026gt;, hostname = esxi03) 24 Component: 7d00ae59-ffba-5b7d-c6cd-0cc47ac2b550 (state = 5, addr space = 214748364800 (200.00GB), \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;disk = 52006064-ff9f-b525-e976-0142b74b4392\u0026lt;/span\u0026gt;, 25 votes = 1, \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;used = 47571795968 (44.00GB), physUsed = 47571795968 (44.00GB)\u0026lt;/span\u0026gt;, hostname = esxi01) 26 Witness: e89eb859-8dac-b7c1-4081-0cc47ac2b550 (state = 5, addr space = 0 (0.00GB), \u0026lt;span style=\u0026#34;color: #0000ff;\u0026#34;\u0026gt;disk = 52b7c9b9-01c1-1770-22af-7f9b78ee3d2f\u0026lt;/span\u0026gt;, 27 votes = 1, used = 4194304 (0.00GB), physUsed = 4194304 (0.00GB), hostname = esxi02) Virtual SAN Default Storage Policy with 100% Object reservation RVC Here we see all parameter values remain the same when compared to the default policy explained above . The only change is “**proportionalCapacity = 100**” , this tells us that the Object space reservation is 100% (Similar to a thick vDisk on a VMFS volumes) /localhost/vSAN6U3-AF/vms\u0026gt; vsan.vm_object_info 21 VM Windows_10: Namespace directory DOM Object: 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452 (v3, owner: 10.109.9.102, policy: spbmProfileGenerationNumber = 0, forceProvisioning = 0, cacheReservation = 0, checksumDisabled = 0, hostFailuresT oTolerate = 1, stripeWidth = 1, spbmProfileId = d4898518-158c-452f-907e-8a3febb0f1a1, proportionalCapacity = [0, 100]) RAID_1 Component: b398a759-303c-a6e3-c09d-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.103, md: naa.55cd2e404c211107, ssd: naa.55cd2e404c20d56d, votes: 1, usage: 0.4 GB) Component: 609eb859-c975-1b18-6be1-0cc47ac2b546 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c212a4c, ssd: naa.55cd2e404c20d56f, votes: 1, usage: 0.4 GB) Witness: 629eb859-8d15-b76e-c30b-0cc47ac2b546 (state: ACTIVE (5), host: 10.109.9.102, md: naa.55cd2e404c210ffd, ssd: naa.55cd2e404c20ce55, votes: 1, usage: 0.0 GB)\nvDISK : Disk backing: [vsanDatastore] 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452/Windows_10.vmdk DOM Object: 1daa5659-ac55-70f6-bc12-0cc47ac2b452 (v3, owner: 10.109.9.102, policy: spbmProfileGenerationNumber = 0, forceProvisioning = 0, cacheReservation = 0, checksumDisabled = 0, hostFailuresT oTolerate = 1, stripeWidth = 1, spbmProfileId = d4898518-158c-452f-907e-8a3febb0f1a1, proportionalCapacity = 100) RAID_1 Component: b898a759-f8d6-0092-b302-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.103, md: naa.55cd2e404c212d55, ssd: naa.55cd2e404c20d56d, votes: 1, usage: 200.0 GB) Component: 7d00ae59-ffba-5b7d-c6cd-0cc47ac2b550 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c211ff5, ssd: naa.55cd2e404c20d56f, votes: 1, usage: 200.0 GB) Witness: e5553f5a-c283-5d3d-e202-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.104, md: naa.55cd2e404c212d05, ssd: naa.55cd2e404c20d047, votes: 1, usage: 0.0 GB) ESXi Output “python /usr/lib/vmware/vsan/bin/vsan-health-status.pyc” 1Namespace directory : 2Object 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452 (v4, owner: esxi02, policy: {\u0026#39;spbmProfileGenerationNumber\u0026#39;: 0, \u0026#39;forceProvisioning\u0026#39;: 0, \u0026#39;cacheReservation\u0026#39;: 0, \u0026#39;checksumDisabled\u0026#39;: 0, \u0026#39;hostFailuresToTolerate\u0026#39;: 1, \u0026#39;stripe 3Width\u0026#39;: 1, \u0026#39;spbmProfileId\u0026#39;: \u0026#39;d4898518-158c-452f-907e-8a3febb0f1a1\u0026#39;, \u0026#39;proportionalCapacity\u0026#39;: [0, 100]}): 4 Configuration 5 RAID_1 6 Component: b398a759-303c-a6e3-c09d-0cc47ac2b452 (state = 5, addr space = 273804165120 (255.00GB), disk = 52ca462c-7d81-3e39-7bcc-858fe854fb81, 7 votes = 1, used = 461373440 (0.00GB), physUsed = 461373440 (0.00GB), hostname = esxi03) 8 Component: 609eb859-c975-1b18-6be1-0cc47ac2b546 (state = 5, addr space = 273804165120 (255.00GB), disk = 52f67161-bba6-9e7c-2a98-79bd160d14e6, 9 votes = 1, used = 452984832 (0.00GB), physUsed = 452984832 (0.00GB), hostname = esxi01) 10 Witness: 629eb859-8d15-b76e-c30b-0cc47ac2b546 (state = 5, addr space = 0 (0.00GB), disk = 52b7c9b9-01c1-1770-22af-7f9b78ee3d2f, 11 votes = 1, used = 4194304 (0.00GB), physUsed = 4194304 (0.00GB), hostname = esxi02) 12 13vDISK : 14 15Object 1daa5659-ac55-70f6-bc12-0cc47ac2b452 (v4, owner: esxi02, policy: {\u0026#39;spbmProfileGenerationNumber\u0026#39;: 0, \u0026#39;forceProvisioning\u0026#39;: 0, 16\u0026#39;cacheReservation\u0026#39;: 0, \u0026#39;checksumDisabled\u0026#39;: 0, \u0026#39;hostFailuresToTolerate\u0026#39;: 1, \u0026#39;stripeWidth\u0026#39;: 1, 17\u0026#39;spbmProfileId\u0026#39;: \u0026#39;d4898518-158c-452f-907e-8a3febb0f1a1\u0026#39;, \u0026#39;proportionalCapacity\u0026#39;: 100}): 18 Configuration 19 RAID_1 20 Component: b898a759-f8d6-0092-b302-0cc47ac2b452 (state = 5, addr space = 214748364800 (200.00GB), disk = 5280ff9a-9880-8549-ad8a-5ee212ee21ee, 21 votes = 1, used = 214752559104 (200.00GB), physUsed = 47571795968 (44.00GB), hostname = esxi03) 22 Component: 7d00ae59-ffba-5b7d-c6cd-0cc47ac2b550 (state = 5, addr space = 214748364800 (200.00GB), disk = 52006064-ff9f-b525-e976-0142b74b4392, 23 votes = 1, used = 214752559104 (200.00GB), physUsed = 47571795968 (44.00GB), hostname = esxi01) 24 Witness: e5553f5a-c283-5d3d-e202-0cc47ac2b452 (state = 5, addr space = 0 (0.00GB), disk = 52eb0e0e-bab1-d12e-4631-df25d1bad218, 25 votes = 1, used = 4194304 (0.00GB), physUsed = 4194304 (0.00GB), hostname = esxi04) vSAN Default Policy with higher stripe width [ ![](/images/understanding-vm-storage-policies-poli-def-sw-4-1-1024x402.png) ](/images/understanding-vm-storage-policies-poli-def-sw-4-1.png) [ ![](/images/understanding-vm-storage-policies-poli-def-sw-4-2-1024x494.png) ](/images/understanding-vm-storage-policies-poli-def-sw-4-2.png) RVC Output **Namespace directory ** DOM Object: 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452 (v3, owner: 10.109.9.102, proxy owner: None, policy: spbmProfileGenerationNumber = 1, forceProvisioning = 0, cacheReservation = 0, checksumDisabled = 0, hostFailuresToTolerate = 1, stripeWidth = 1, spbmProfileId = d4898518-158c-452f-907e-8a3febb0f1a1, proportionalCapacity = [0, 100]) RAID_1 Component: b398a759-303c-a6e3-c09d-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.103, md: naa.55cd2e404c211107, ssd: naa.55cd2e404c20d56d, votes: 1, usage: 0.4 GB, proxy component: false) Component: 609eb859-c975-1b18-6be1-0cc47ac2b546 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c212a4c, ssd: naa.55cd2e404c20d56f, votes: 1, usage: 0.4 GB, proxy component: false) Witness: 629eb859-8d15-b76e-c30b-0cc47ac2b546 (state: ACTIVE (5), host: 10.109.9.102, md: naa.55cd2e404c210ffd, ssd: naa.55cd2e404c20ce55, votes: 1, usage: 0.0 GB, proxy component: false)\nvDISK : Disk backing: [vsanDatastore] 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452/Windows_10.vmdk DOM Object: 1daa5659-ac55-70f6-bc12-0cc47ac2b452 (v3, owner: 10.109.9.102, proxy owner: None, policy: spbmProfileGenerationNumber = 1,forceProvisioning = 0,cacheReservation = 0, checksumDisabled = 0, hostFailuresToTolerate = 1, stripeWidth = 4, spbmProfileId = d4898518-158c-452f-907e-8a3febb0f1a1, proportionalCapacity = 0)\nRAID_1 RAID_0 Component: b550405a-258f-8b6e-eda1-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c212bd1, ssd: naa.55cd2e404c20d56f, votes: 3, usage: 11.1 GB, proxy component: false) Component: b550405a-81c0-8e6e-ae04-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.103, md: naa.55cd2e404c211107, ssd: naa.55cd2e404c20d56d, votes: 4, usage: 11.1 GB, proxy component: false) Component: b550405a-0ec8-906e-314a-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.101, md: naa.55cd2e404c2110e4, ssd: naa.55cd2e404c20d56f, votes: 2, usage: 11.1 GB, proxy component: false) Component: b550405a-6add-926e-7200-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.102, md: naa.55cd2e404c2129ec, ssd: naa.55cd2e404c20ce55, votes: 4, usage: 11.1 GB, proxy component: false) RAID_0 Component: b550405a-15fe-946e-7ed6-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.104, md: naa.55cd2e404c212d08, ssd: naa.55cd2e404c20d047, votes: 1, usage: 11.1 GB, proxy component: false) Component: b550405a-dbe6-966e-ebf3-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.104, md: naa.55cd2e404c2110eb, ssd: naa.55cd2e404c20d047, votes: 1, usage: 11.1 GB, proxy component: false) Component: b550405a-9575-986e-7fcd-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.104, md: naa.55cd2e404c212d05, ssd: naa.55cd2e404c20d047, votes: 1, usage: 11.1 GB, proxy component: false) Component: b550405a-922a-9a6e-504d-0cc47ac2b452 (state: ACTIVE (5), host: 10.109.9.104, md: naa.55cd2e404c212d4e, ssd: naa.55cd2e404c20d047, votes: 1, usage: 11.1 GB, proxy component: false)\nESXCLI Output Namespace directory\nObject 1aaa5659-6b1a-a7ba-05dd-0cc47ac2b452 (v4, owner: esxi02, policy: {'spbmProfileGenerationNumber': 1, 'forceProvisioning': 0, 'cacheReservation': 0, 'checksumDisabled': 0, 'hostFailuresToTolerate': 1, 's tripeWidth': 1, 'spbmProfileId': 'd4898518-158c-452f-907e-8a3febb0f1a1', 'proportionalCapacity': [0, 100]}): Configuration RAID_1 Component: b398a759-303c-a6e3-c09d-0cc47ac2b452 (state = 5, addr space = 273804165120 (255.00GB), disk = 52ca462c-7d81-3e39-7bcc-858fe854fb81, votes = 1, used = 461373440 (0.00GB), physUsed = 461373440 (0.00GB), hostname = esxi03) Component: 609eb859-c975-1b18-6be1-0cc47ac2b546 (state = 5, addr space = 273804165120 (255.00GB), disk = 52f67161-bba6-9e7c-2a98-79bd160d14e6, votes = 1, used = 452984832 (0.00GB), physUsed = 452984832 (0.00GB), hostname = esxi01) Witness: 629eb859-8d15-b76e-c30b-0cc47ac2b546 (state = 5, addr space = 0 (0.00GB), disk = 52b7c9b9-01c1-1770-22af-7f9b78ee3d2f, votes = 1, used = 4194304 (0.00GB), physUsed = 4194304 (0.00GB), hostname = esxi02)\nvDISK :\nObject 1daa5659-ac55-70f6-bc12-0cc47ac2b452 (v4, owner: esxi02, policy: {'spbmProfileGenerationNumber': 1, 'forceProvisioning': 0, 'cacheReservation': 0, 'checksumDisabled': 0, 'hostFailuresToTolerate': 1, 'stripeWidth': 4, 'spbmProfileId': 'd4898518-158c-452f-907e-8a3febb0f1a1', 'proportionalCapacity': 0}): Configuration RAID_1 RAID_0 Component: b550405a-258f-8b6e-eda1-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 52ad1443-71bd-84fc-20c4-024b8e260ee4, votes = 3, used = 11941183488 (11.00GB), physUsed = 11941183488 (11.00GB), hostname = esxi01) Component: b550405a-81c0-8e6e-ae04-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 52ca462c-7d81-3e39-7bcc-858fe854fb81, votes = 4, used = 11949572096 (11.00GB), physUsed = 11949572096 (11.00GB), hostname = esxi03) Component: b550405a-0ec8-906e-314a-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 5218da7e-cf6c-35d4-2dcf-215b93249581, votes = 2, used = 11936989184 (11.00GB), physUsed = 11936989184 (11.00GB), hostname = esxi01) Component: b550405a-6add-926e-7200-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 52aeebab-02c1-4425-ac85-399569d4fec9, votes = 4, used = 11941183488 (11.00GB), physUsed = 11941183488 (11.00GB), hostname = esxi02) RAID_0 Component: b550405a-15fe-946e-7ed6-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 525ad3e9-40dc-aa54-9cc9-4bc6fa70aa15, votes = 1, used = 11941183488 (11.00GB), physUsed = 11941183488 (11.00GB), hostname = esxi04) Component: b550405a-dbe6-966e-ebf3-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 524392d2-4987-5264-fbea-358cc6013aef, votes = 1, used = 11949572096 (11.00GB), physUsed = 11949572096 (11.00GB), hostname = esxi04) Component: b550405a-9575-986e-7fcd-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 52eb0e0e-bab1-d12e-4631-df25d1bad218, votes = 1, used = 11936989184 (11.00GB), physUsed = 11936989184 (11.00GB), hostname = esxi04) Component: b550405a-922a-9a6e-504d-0cc47ac2b452 (state = 5, addr space = 53687091200 (50.00GB), disk = 5278dc1c-85d6-49ae-0779-9a67e20c3109, votes = 1, used = 11941183488 (11.00GB), physUsed = 11941183488 (11.00GB), hostname = esxi04)\n","link":"https://blogs.hareeshkg.bid/post/understanding-vm-storage-policies/","section":"post","tags":[],"title":"Understanding VM-Storage-Policies"},{"body":"New “Cross vCenter Workload Migration Utility” which is GUI based is now available on the VMware-labs\nThis is a boon to many of the administrators , who were earlier using the CLI utility or even the Regular vCenter servers in the same SSO domain . This new tool will allow administrators to XVmotion the VMs between vcenter servers that are not in the same SSO Domain . Also allows XVMotion between a vcenter running any flavour of 6.0 to a 6.5 vcenter server . However XVmotion doesnt work the other away around that is between a 6.5 vCenter to another 6.0 Vcenter server . All XVmotion between the same releases should work smoothly .\nSteps to download and install / usage is available under instructions , tried them out *ONLY on your labs not production its officially not supported * . Here is my first impressions\n1.Download and launch the tool using cmd prompt . 2. Used the Port 31000 on the local machine , I launched the tool . 3. Launch the Browser , on the same local machine where you had installed the tool , i.e http://localhost:31000 , you will be in the landing page. 4. Now I added two vcenter server to kick off an xVmotion task , on the landing page select : Migrate ⇒ Register , you will be asked to fill up your site deteails , vcenter hostname/user/password and submit . Follow the same proceed to add both your source and destination vcenter server . Here I chose a 6.0 vcenter as my source and a 6.5U1c Vcenter as my destination (Note**: These are not in the same SSO Domain) If there were issues while registering your vCenters , see the cmd-prompt logging , find out the error .\nEx :\n01:02:41 INFO Registering site 6.5Vcenter-vSAN for vc: 10.109.44.29 01:02:51 ERROR Could not connect to vcenter 10.109.44.29 com.vmware.xvm.XvmException: com.vmware.vim25.InvalidLoginFaultMsg: Cannot complete login due to an incorrect user name or password.\n[ ![](/images/new-cross-vcenter-workload-migration-utility-XVMOTION-4-1024x527.png) ](/images/new-cross-vcenter-workload-migration-utility-XVMOTION-4.png) 5. Initiate your migration . NOTE***: **Do not use a DOT”.” on your site name , you will encounter on Error fetching site details when you choose the source or destination while initiating XVmotion . NOTE***: **You must have Cross vMotion license between the source hosts and destination host , other wise you may see an Error at the source vcenter for licensing . ","link":"https://blogs.hareeshkg.bid/post/new-cross-vcenter-workload-migration-utility/","section":"post","tags":[],"title":"New Cross vCenter Workload Migration Utility"},{"body":"Hello all here is a table to compare the versions of different VxRAIL manager with VCenter server / vSAN and ESXI build . If you were planning to upgrade your vcenter /ESXI or vSAN release to address any known problems , you will have to upgrade this from the VxRAIL manager and not directly thru VC/ESXi . Please engage your EMC support to assist you with the upgrade process .\nHere is a table of matching VxRAIL and VMware release . This information is now available thru VMware official KB : [https://kb.vmware.com/s/article/52075\n](https://kb.vmware.com/s/article/52075)\nI recommend to see VxRAIL-Official-compatibility-Matrix to get updates on the latest releases .\nVxRail Release System Version Esxi (Version – Build #) VxRail manager vSAN Internal vCSA External vCSA Min |Recomm | Max\n3 3.0.0.3482248 6.0 U1b – 3380124 3.0.0.3482248 (VRMe) 6.1 6.0 U1b N/A N/A N/A 3.5 3.5.0.3936871 6.0 U2 – 3620759 3.5.0.3936871 6.2 6.0 U2 6.0 U2 6.0 U2 or later 6.0 U3b 4 4.0.0-4631185 6.0 U2 P03 – 4192238 4.0.0-4631184 6.2 6.0 U2 6.0 U2 6.0 U2 or later 6.0 U3b 4.0.00001 4.0.00001-4631185 6.0 U2 P04 – 4600944 4.0.0-4631184 6.2 6.0 U2 6.0 U2 6.0 U2a or later 6.0 U3b 4.0.100 4.0.100-4907697 6.0 U2 P04 – 4600944 4.0.100-4907696 6.2 6.0 U2a 6.0 U2 6.0 U2a or later 6.0 U3b 4.0.132 4.0.132.00000-5128178 6.0 U3 – 5050593 4.0.132-5128177 6.2 6.0 U3 6.0 U2 6.0 U3 or later 6.0 U3b 4.0.200 4.0.200-5234143 6.0 EP07a – 5224934 4.0.200-5234142 6.2 6.0 U3a 6.0 U2 6.0 U3 or later 6.0 U3b 4.0.300 4.0.300-5629429 6.0 P5 – 5572656 4.0.300-5629428 6.2 6.0 U3b 6.0 U3 6.0 U3b or later 6.0 U3b 4.0.301 4.0.301-6026519 6.0 P5 – 5572656 4.0.301-6026518 6.2 6.0 U3b 6.0 U3 6.0 U3b or later 6.0 U3b 4.0.302 4.0.302-6829778 6.0 EP11 – 6765062 4.0.302-6829777 6.2 6.0 U3b 6.0 U3 6.0 U3b or later 6.0 U3b 4.0.310 4.0.310-6137115 6.0 P5 – 5572656 4.0.310-6137114 6.2 6.0 U3b 6.0 U3 6.0 U3b or later N/A 4.0.400 4.0.400-7209247 6.0 P6 – 6921384 4.0.400-7209246 6.2 6.0 U3c 6.0 U3c N/A 4.5 4.5-6607744 6.5 U1 – 5969303 4.5-6607743 6.6.1 6.5 U1 6.5 U1 6.5 U1 N/A 4.5.070 4.5.070-6881819 6.5 EP02 – 6765664 4.5.070-6881818 6.6.1 6.5 U1a 6.5 U1 6.5 U1a 6.7GA 4.5.211 4.5.211 8667744 6.5 U2 8294253 4.5.211 8667744 6.6.1U2 6.5 U2 8307201 6.5U2 6.5 U2 6.5U2b 4.5.212 4.5.212 8993024 6.5 EP07 8935087 4.5.212 8993024 6.6.1 U2 6.5 U2b 8815520 6.5U2b 6.5 U2b 6.5U2b ","link":"https://blogs.hareeshkg.bid/post/correlating-vxrail-release-with-vmware-build-numbers/","section":"post","tags":[],"title":"Correlating VXRAIL release with VMware Build Numbers"},{"body":"To understand vSAN component placement and sizing , we should be proficient in creating storage policy and understand the parameters you choose while creating a storage policy . I recommend you to read the vm-storage-policy guide before proceeding .\nPlease note the following Section “Understanding Component Placements” will not apply to vSAN Clusters running version 7 U1, as there are some changes for placements, which considers new space efficiency feature for slack-space. Please read the new blog for “Component-Placement-7U1 and above”.\nTypes of Objects seen in a vSAN environment A VM can have five different types of objects on a vSAN datastore as follows, keeping in mind that each VM may have multiples of some of these objects associated with it: VM home or “namespace directory” A swap object (if the VM is powered on) Virtual disks/VMDKs Delta disks (each an object) created for snapshots Snapshot memory (each an object) optionally created for snapshots The VM namespace may need a little further explanation. All VMs files, excluding VMDKs, deltas (snapshots), memory (snapshots) and swap, reside in an area called the VM namespace on vSAN. The typical files found in the VM home namespace are the .vmx, the .log files, .vmdk descriptor files, and snapshot deltas descriptors files and everything else one would expect to find in a VM home directory.\nEach storage object is deployed on vSAN as a RAID tree, and each leaf of the tree is said to be a component. For instance, if I choose to deploy a VMDK with a stripe width of 2, but did not wish to tolerate any failures (for whatever reason), a RAID-0 stripe would be configured across a minimum of two disks for this VMDK. The VMDK would be the object, and each of the stripes would be a component of that object.\nSimilarly, if I specified that my VMDK should be able to tolerate at least one failure in the cluster (host, disk, or network), and left all the other policy settings at their defaults, a RAID-1 mirror of the VMDK object would be created with one replica component on one host and another replica component on another host in my vSAN cluster. Finally, if my policy included a requirement for both striping and availability, my striped components would be mirrored across hosts, giving me a RAID 0+1 configuration. This would result in four components making up my single object, two striped components in each replica.\nNote that delta disks are created when a snapshot is taken of a VM. A delta disk inherits the same policy as the parent disk (stripe width, replicas, and so on).\nThe swap object is created only when the VM is powered on.\nThere is another component called the witness. The witness component is very important and special. Although it does not directly contribute toward VM storage, it is nonetheless an important component required to determine a quorum for a VM’s storage objects in the event of a failure in the cluster. We will return to the witness component shortly, but for the moment let’s concentrate on VM storage objects.\nTo Understand how to view component placements for a VM or an Object , see : Link\nUnderstanding Component placements Now lets understand how the components are placed the way it is placed whenever you create a VM or a VMDK for a VM . Remember this is controlled and dictated by the policy defined for the VM or VMDK that you are about to create . Here is a small set of illustrations of different types of Object placement without considering witness objects , since we know by now that it is generally used as Quorum /Tie-breaker and this will be placed on of the hosts which should contain data or mirror components .\nGlossary :\nFTT = Failures to Tolerate ( generally host/disk/diskgroup )\nSW = Stripe width (stripped component for an Object , note that if we dont have enough disks to satisfy the SW defined , vsan can still create two stripes on one disk within a Disk group)\nThe following Illustrations explain how components are placed with RAID-0 and RAID-1 as a part of default sets of VM storage policy which is applicable for both Hybrid and All-Flash setup .\nPolicy of FTT= 0 and SW =1 created, Policy of FTT=1 and SW=1 FTT=2 and SW=1\nFTT=1 and SW=2\nFTT=2 and SW=3\nFTT=1 and SW=1 , Object size =400GB (larger than 255GB) .\nFTT=0 and SW=1 , Object size = 600GB (larger than 255GB) , here choosing sw=3 or sw=1 does not make a difference we will still create three components\nWhat happens when we change the SW for an objects thru storage policy Ftt=1 and SW=2 to FTT=1 and SW=3 . For a brief period of time , double the storage capacity will be consumed by this object until new object/component creations complete , post which original components and objects are deleted .\nLets now see how components are placed with RAD5/6 erasure coding with space efficiency .\nFor RAID-5 Objects : **Limitations : **FTT is fixed to 1 , Minimum hosts required is 3+1 (4 node cluster) , Only available for all-flash vSAN configurations\n**Benefits : **Saves space compared to traditional RAID1/0 approach with the use of distributed parity blocks.\nFTT=1 , SW=1 FTT=1 , SW=2\nFor RAID-6 Objects :\n**Limitations : **FTT is fixed to 2 , Minimum hosts required is 4+2 (6 node cluster) , Only available for all-flash vSAN configurations **Benefits : **Saves space compared to traditional RAID1/0 approach with the use of distributed parity blocks.\nFTT=2 , SW=1 FTT=2 , SW=2\n","link":"https://blogs.hareeshkg.bid/post/understanding-vsan-objects-and-components/","section":"post","tags":[],"title":"Understanding vsan Objects and Components"},{"body":"“VxRAIL Hosts going Non-responsive” symptoms are seen because of many reason , here we are going to discuss about one such issues seen which is very common lately .\nThe hosts Generally go non-responsive is because the hostd service on the ESXi hosts is non-responsive . The fastest way to fiund is through the iDRAC / KVM console to the hosts directly and press ALT+F12(Function Key 12) or F11 (Function Key 12)on your keyboard to see live vmkernel logging and you will see messages relating to the hostd service going non-responsive , further if SSH is available to the hosts we should be able to check the logs live .\nLately we have been noticing that a VxRAIL environment running DELL hardware are running into this issue because of a plugin conflict which locks up the esx.conf file (ESXI config file) and doesn’t release this lock and eventually causing the hostd service going non-responsive . The conflict is between Dell-PTA AGENT and lsu-lsi-lsi-msgpt3-plugin .\nThis known issue with lsu-lsi-lsi-msgpt3-plugin and Dell-PTA AGENT vibs installed on the VXRAIL Boxes causes a lock on the esx.conf file intern causing the hosts to go non-responsive . The root cause is clearly documented on EMC KB : https://emcservice.force.com/CustomersPartners/kA2f1000000FvH4CAK .\n**Cause : **“esxcfg-mpath” related commands call lsu plugin init function, which calls StoreLib looking for LSI controllers. StoreLib sends a lot of\nIOCTL command to driver. In an escalated case, one command hang at HW side for 129 seconds, it holds the lock for /etc/vmware/esx.conf, the next command waiting to request the esx.conf.LOCK eventually leads to ESXi host hang and non-responsive. Removing the lsu-lsi-lsi-msgpt3-plugin could mitigate the issue.\nIt is recommend to un-install the “lsu-lsi-lsi-msgpt3-plugin” proactively even though if you have not hit this issue and if both these plugin are present . If you dont have access to this link , find the steps below\nConflicting Drivers found installed :\nesxcli software vib list | grep -i “lsu-lsi-lsi-msgpt3” \u0026amp;\u0026amp; esxcli software vib list | grep -i “dellptagent “ lsu-lsi-lsi-msgpt3-plugin 1.0.0-1vmw.600.0.0.2494585 VMware VMwareCertified 2017-07-05\ndellptagent 1.1-0.5885 Dell PartnerSupported 2017-07-05\nLog Evidence for the issue : 1**vmkernel.log:2017-12-01T06:49:07.114Z cpu24:49140)FSS: 6264: Failed to open file \u0026#39;naa.5000cca0805ff528\u0026#39;; Requested flags 0x5, world: 49140 [DellPTAgent], (Existing flags 0x4005, world: 35442 [hostd-worker]): Busy** 2**vmkernel.log:2017-12-01T06:52:12.662Z cpu18:49140)FSS: 6264: Failed to open file \u0026#39;naa.5000cca0805ff264\u0026#39;; Requested flags 0x5, world: 49140 [DellPTAgent], (Existing flags 0x4005, world: 38219 [hostd-worker]): Busy** 3**vmkernel.log:2017-12-01T06:54:22.738Z cpu7:49140)FSS: 6264: Failed to open file \u0026#39;naa.5000cca0805ff264\u0026#39;; Requested flags 0x5, world: 49140 [DellPTAgent], (Existing flags 0x4005, world: 35927 [hostd-worker]): Busy** 4 5**\u0026lt;span style=\u0026#34;color: #ff0000;\u0026#34;\u0026gt;vmkernel.1:2017-12-01T03:17:29.546Z cpu40:194056575)ALERT: hostd detected to be non-responsive\u0026lt;/span\u0026gt;** 6**\u0026lt;span style=\u0026#34;color: #ff0000;\u0026#34;\u0026gt;vmkernel.1:2017-12-01T03:18:33.229Z cpu38:200259473)ALERT: hostd detected to be non-responsive\u0026lt;/span\u0026gt;** 7 8**hostd.log:2017-11-30T21:24:54.179Z warning hostd[A42C1B70] [Originator@6876 sub=Hostsvc.NetworkProvider opID=1313cfa6-43ec user=vpxuser] Error getting dvs fa 3f 02 50 f3 c3 f2 a7-bd 52 85 de 45 c5 3b d5 : Error interacting with configuration file /etc/vmware/esx.conf: Timout while waiting for lock, /etc/vmware/esx.conf.LOCK, to be released.\u0026lt;span style=\u0026#34;color: #ff0000;\u0026#34;\u0026gt; Another process has kept this file locked for more than 30 seconds. The process currently holding the lock is esxcfg-mpath(198411757). This is likely a temporary condition. Please try your operation again.\u0026lt;/span\u0026gt;** 9\u0026lt;span style=\u0026#34;color: #ff0000;\u0026#34;\u0026gt;**hostd.log:--\u0026gt; value = \u0026#34;Error interacting with configuration file /etc/vmware/esx.conf: Timout while waiting for lock, /etc/vmware/esx.conf.LOCK, to be released. Another process has kept this file locked for more than 30 seconds. The process currently holding the lock is esxcfg-mpath(198411757). This is likely a temporary condition. Please try your operation again.\u0026#34;**\u0026lt;/span\u0026gt; Step1 : Place one host in maintenance-mode with ensure accessibility Please also see : Link which explains few basic checks before placing hosts in maintenance Mode .\nStep2 : Remove the problematic vib esxcli software vib remove –no-live-install -n lsu-lsi-lsi-msgpt3-plugin\nRemoval Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true VIBs Installed: VIBs Removed: VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-1vmw.600.0.0.2494585\nStep3 : Reboot the host Post rebooting the host wait for host to come back in maintenanceMode wait for any resync , you may use the webclient on vcenter Select : Cluster ⇒ Monitor ⇒ vsan ⇒ Resyncing Components or use RVC command : ” vsan.resync_dashboard .” (see how-to-login to RVC) under cluster directory . If there is an ongoing resync , DONOT attempt maintenance on any of the other hosts , wait for the resync to complete . If there were no resync operations seen , proceed to perform same steps with all other hosts by placing them in maintenance remove the vib and rebooting them one at time .\n","link":"https://blogs.hareeshkg.bid/post/vxrail-hosts-going-non-responsive-due-to-a-plugin-conflict/","section":"post","tags":[],"title":"VxRAIL Hosts going Non-responsive due to a plugin conflict"},{"body":"At many times we may wonder while managing multiple vSAN clusters in a production environment , what are my”Things to know for vSAN management” you may ask yourself ?,a Complete guide for your every day tasks while managing vSAN clusters and things to know which can cause potential disaster without taking precautions\nVirtual machine storage policy compliance You might have already seen all my other previous hosts and by now you would have realized that virtual machine storage policy is everything when it comes to data protection . We need to always make sure all the important virtual machines are compliant with their storage policies at all times and if there were any VMs running FTT-0 which we may still need redundancy , change the policy to FTT-1 (fault to tolerate). Please see How-to-create-VM-policy-vSAN .\nAlways make a habit to read release notes and patch information for vSAN It is highly recommended to know what has come? and what is coming ? To stay on top of things and avoid major/minor issues in the environment,hence always make a habit to read the field advisory emails that we receive from when new patches updates are received at least the **critical **ones . From my experience so for in handling vSAN issues vs fixes , any version running ESXi 6.0 update 2 and below needs immediate upgrade to the latest version ESXi 6.0 update 3 patch 06 or if the hardware is compatible with vSAN 6.6 (ESXi 6.5 Update 1+) we should go for this . There are always enhancements , fixes for certain known issues in the latest release and few may be nasty issues which we always want to avoid .\nWe know upgrades are always painful as we may be involved with multiple products and dependencies with other suites (VC , SRM , Horizon and even ready appliances like VXRAIL,VXRACK SDDC). Hence we can plan the upgrades well do the Pre-check with all vendors and suites for compatibility /interoperability . In many cases it might just be a patch to the current version level and may not require a version upgrade so in these situations we may not necessarily upgrade other product suites and the update process would be a hassle free . See below KB articles for awareness .\nFollowing KBs are applicable for both Hybrid and All-Flash setup\n**ESXi 6.0 Patch 06 ** KB2146345 – ESXi host experiences a PSOD due to a vSAN race condition\nKB2145347 – Component metadata health check fails with invalid state error\nKB2150189 – vSAN de-staging may cause a brief PCPU lockup during heavy client I/O\nKB2150395 – Bytes to sync values for RAID5/6 objects appear incorrectly in vCenter and RVC\nKB2150396 – Using objtool on a vSAN witness node may result in a PSOD\nKB2150390 – Health check for vSAN vmknic configuration may display a false positive\nKB2150389 – SSD congestion may cause multiple virtual machines to become unresponsive\nKB2150387 – vSAN Datastores may become inaccessible during log or memory congestion\nKB2151127 – vsan and Vmware Bootbank critical fix\nKB2151132 – vsan and Vmware Bootbank critical fix\nFollowing KBs are Highly Recommended for All-Flash setup .\n**ESXi 6.5 Express patch 4 : **Critical Fix for ALL Flash KB Running 6.5 releases **ESXi 6.0 Express patch 11 : **Critical Fix for ALL Flash KB Running 6.0 releases Note : You may need to contact your hardware vendor while using an appliance like DELL-EMC (VXRAIL , VXRACK) , IBM-Bluemix etc.. prior to applying patches*\nvSAN Health check plugin It is highly recommended to monitor the vSAN health check plugin under (Cluster ⇒ Monitor ⇒ vSAN) for all your clusters for any issues with vSAN . Most of the alerts are self explanatory with proper description , if you needed further assistance in troubleshooting the issue , feel free to contact Technical support or use Ask VMware button .\nHandling and troubleshooting Non-responsive Hosts At many times the admins tend to reboot a hosts or even multiple hosts to get the hosts responsive on a vcenter for management . You should highly refrain from taking such actions unless you know what yu are doing , such actions can cause “Data Unavailability and Data Loss ” situations . It is better to contact vSAN Technical support prior to taking such actions . Here is an example one host running into high congestion (Log/SSD/etc..) causes multiple or all hosts to go non-responsive in the cluster and addressing such one host may get all other hosts responding . We need to take preventive measures just to reboot one host . Hence do not panic and reboot hosts as you may end up in a bigger problem to deal with .\n*Tip : ** It is good to have SSH turned on for all hosts running vSAN , unless company policy dictates for a reason not to . It is easy to isolate the problem to a node or objects when we have a SSH to the hosts and looking at live vmkernel.log /vobd.log /hostd.log will surely give us something .\nHow and when to change the storage policy for the VMs You may be in need to change the storage policy for all VMs at a time or some of the Vms . You may wonder will there be any consequences doing this , well YES you might just take down the whole cluster!! . Now here is the explanation , if we change a policy for a VM (EG : FTT=1 SW=1 “Default vSAN Policy” to FTT=1 SW=2 “Custom Default Policy”) , in this case vSAN creates a top-level RAID-1 objects starts to create the components for the new Policy and once the components are successfully created and synced , the older components are cleaned up (Deleted).\nThis means that if we have a VM of size 5TB (current allocation) and you just applied a new polity to it , we would be utilizing an additional 5TB of storage space on vSAN while transition is in progress , hence ensure to have enough headroom on vSAN Datastore while changing storage policy for one or more VMs. Also note that you will be triggering resyncs for every policy change and further cause performance issues and congestion on hosts if there were too many re-syncing objects (VMs) . See How-to-change-policy\nBest Practices for storage policy change Make sure you have enough space to accommodate the additional consumption during the re-sync progress and with a good head room (VM size + a head room) .\nMake sure there are no previous huge resync prior to triggering a policy change .\nMake sure there are no congestion issues / warnings against the host prior to triggering a policy change (see vSAN health plugin).\nEnsure all the capacity disks on your disk groups are balanced out , else run a proactive re-balance and then attempt any policy change . Please us RVC commands to achieve this\nDo not Trigger policy change for too many VMs at one go as a best practice .\nIf there were any VMs running into performance issues while this policy change is in progress , reduced the resync threshold (from available GUI button with vSAN 6.6) , else contact support for recommendations .\nDisk Group Maintenance Activities If there comes a situation to delete disk group for whatever reason , it is recommended to perform a pre-check on the cluster before attempt a disk-group deletion activities. Follow steps would help you in doing some pre-check . Deleting a disk group can cause potential data loss and data unavailability (DU/DL) if you don’t understand what you were doing .\nUnderstand the number of nodes that you have in the cluster and correct choice of modes to delete a disk group on your cluster .\nIf you were running a third node VSAN setup , the only two available options are (no data migration and ensure accessibility mode) .\nYou have to make sure that there were no vSAN objects or virtual machines running with a policy of FTT-0 (RAID-0) on a DG which is about to be deleted with no actions . This is an irreversible options , hence do such tasks are your own risk .\nIf were going to choose ensure accessibility as an option to complete the task , you are at risk of loosing good data objects /components if something were to happen to the other hosts and diskgroups when this DG is being deleted . Remember “Always take backups” before taking any disruptive actions like this.\nWhen we talk bout Full Data Migration /Evacuation , we need to take additional Pre-cautions when it comes to an ALL-FLASH setup with dedup and compression enabled cluster . Please remember all Dedup and compression tasks happen within a Disk group and not even across other disk groups within a host . Hence when you delete a diskgroup with full data-Migration/Evacuation you will have a resync for uncompressed /un-deduped data which may be huge depending on your previous de-dup and compression ratio achieved on that DG which is being deleted . Make sure you have enough space (equivalent and higher than the actual disk group) that was actually deleted to avoid any cluster wide problems (DU/DL) .\n","link":"https://blogs.hareeshkg.bid/post/things-to-know-for-vsan-management/","section":"post","tags":[],"title":"Things to know for vSAN management"},{"body":"How to silence warnings on vSAN health Plugin? There are many situations that you may be aware of a particular warning which is not very critical and which is something you are already aware of , however you would like to have the warning silenced or disabled and would like to see the health check come clean every-time time you log in the web-client . There may be known warnings on the vSAN health check plugin you may see in a vSAN cluster , Ex : vMotion: MTU check (ping with large packet size) , vSAN Build Recommendation Engine Health , Component metadata health, Customer experience improvement program (CEIP) and vSAN HCL DB Auto Update ..etc.\nComponent metadata health check fails with invalid state error on vSAN? You may be wondering if there is a way we can disable a certain which you are already aware of and which should not be impacting your environment such as ” Component metadata health” . This issue was only seen with vSAN release 6.2 running version be ESXi 6.0 Update3 Patch 05 (6.0 Update 3a) , please see KB:Component-Meta-Data-Health_check-Fail\nThe above error is only thrown on stale components that reside on the vSAN datastore and doesnot essentially impact any of your VM or VM objects , these are stale /left over objects which are on certain disks in a diskgroup and generally go away when you delete and re-create the disk , however this is an unnecessary task and will be a very costly operation when it comes to all flash setup as we cannot delete an individual disk and whole DG will be needing a deletion and re-creation . This will trigger a very huge resync , possibly overrun your vSAN network bandwidth which can cause impact to your production VMs, it is okay to ignore this error .\nSilence a check or warning on vSAN health Plugin We can use vSAN RVC client to disable certain checks on vsan so that we donot see the warning on webclient going forward .\nStep1 : Log into RVC console . See how-to-login -to-RVC Step2 : Navigate to your respective vSAN cluster for which we would like to silence /suppress /disable a warning . We will be using commands “vsan.health.silent_health_check_status” and “vsan.health.silent_health_check_configure” Command Usage :\nA) /localhost/vsan66-AF-DELL/computers/vsan66-ALL-FLASH-DELL\u0026gt; vsan.health.silent_health_check_status --help usage: silent_health_check_status cluster... Get the current silent health check list for the given cluster cluster: Path to a ClusterComputeResource -h, --help Show this message B) /localhost/vsan66-AF-DELL/computers/vsan66-ALL-FLASH-DELL\u0026gt; vsan.health.silent_health_check_configure --help usage: silent_health_check_configure [opts] cluster Configure silent health check list for the given cluster cluster: Path to a ClusterComputeResource -a, --add-checks= Add checks to silent list, usage: -a [Health Check Id]. -r, --remove-checks= Remove checks from silent list, usage: -r [Health Check Id]. To restore the silent check list, using '-r all' -i, --interactive-add Use interactive mode to add checks to the silent list -n, --interactive-remove Use interactive mode to remove checks from the silent list -h, --help Show this message\nTo see list of all available health checks , their IDs and current silience status , run , run command “**vsan.health.silent_health_check_status .**” Step3 : To disable a certain a health check , here in this case it is “componentmetadata” . We may silence this warning by either using the interactive method using switch -i (–interactive) or by using switch -a (–add-checks).\nTo disable using interactive Method run command “vsan.health.silent_health_check_configure . -i” , in our case the number is 58 which we need to enter for component metadata health check to be disabled .\nStep4 : Verify and confirm if component Meta data health is silenced by running “vsan.health.silent_health_check_status .” and webclient. 1[ 2![](/images/how-to-silence-warnings-on-vsan-health-plugin-vsan-slience-health-check-usage-3-150x150.png) 3](/images/how-to-silence-warnings-on-vsan-health-plugin-vsan-slience-health-check-usage-3.png)[ 4![](/images/how-to-silence-warnings-on-vsan-health-plugin-vsan-slience-health-check-usage-4-150x150.png) 5](/images/how-to-silence-warnings-on-vsan-health-plugin-vsan-slience-health-check-usage-4.png) ","link":"https://blogs.hareeshkg.bid/post/how-to-silence-warnings-on-vsan-health-plugin/","section":"post","tags":[],"title":"How to Silence Warnings on vSAN health Plugin"},{"body":"How to Login into RVC? , I have been asked this question multiple times and there is no default KB and direct link explaining the steps to login to RVC for vCenter servers . This article explains steps to login into RVC for both vcenter Server Appliance and Windows based vcenter server .\nHow to log into RVC on a VCSA (appliance) Step1 : Login to VCSA with root credentials . Example :\nUsing username “root”.\nVMware vCenter Server Appliance 6.5.0.10000\nType: vCenter Server with an external Platform Services Controller\nLast login: Sat Sep 30 12:11:09 2017 from 10.104.1.23\nroot@localhost [ ~ ]#\nStep2 : Type command “rvc” (without quotes) Example :\nroot@localhost [ ~ ]# rvc Install the \u0026quot;ffi\u0026quot; gem for better tab completion. Host to connect to (user@host):\nStep3 : You will be prompted for user@host , here you will need to use a user who has administrator privilages on vCenter ,vSAN Datacenter and vSAN clusters . Example : Using Administrator@vsphere.local , I logged into the same vCenter by typing administrator@vsphere.local@localhost , you may also try logging into a different vcenter from a different VC like : administrator@vsphere.local@182.168.1.7 root@localhost [ ~ ]# rvc Install the \u0026quot;ffi\u0026quot; gem for better tab completion. Host to connect to (user@host): administrator@vsphere.local@localhost password: 0 / 1 localhost/\nStep4 : You will now need to understand if you were going to run Cluster level / host level / VM level commands and you need to be in respective directory while run commands specific to Cluster/Hosts/VMs . Please cd to the respective directory and then try a command . Note* : Use ls to see list of tree directory under each directory .\nExample :\ncd 1 /localhost\u0026gt; ls 0 vsan66-AF-DELL (datacenter) /localhost\u0026gt; cd 0 /localhost/vsan66-AF-DELL\u0026gt; ls 0 storage/ 1 computers [host]/ 2 networks [network]/ 3 datastores [datastore]/ 4 vms [vm]/ /localhost/vsan66-AF-DELL\u0026gt; cd 1 /localhost/vsan66-AF-DELL/computers\u0026gt; ls 0 Strect-66-virtual (cluster): cpu 119 GHz, memory 14 GB 1 vsan66-ALL-FLASH-DELL (cluster): cpu 100 GHz, memory 327 GB 2 10.109.32.176 (standalone): cpu 2 GHz, memory 3 GB /localhost/vsan66-AF-DELL/computers\u0026gt; cd 1 /localhost/vsan66-AF-DELL/computers/vsan66-ALL-FLASH-DELL\u0026gt;\nStep5 : In the above example you navigated from vCenter⇒Datacenter⇒Clusters⇒vSANCluster . Now you are good to run any of the vSAN RVC commands here which applies to cluster level . Please see list of all RVC commands here Example for a cluster level command for check Re-syncing objects on cluster :\n/localhost/vsan66-AF-DELL/computers/vsan66-ALL-FLASH-DELL\u0026gt; vsan.resync_dashboard . 2017-09-30 13:04:35 +0000: Querying all VMs on vSAN ... 2017-09-30 13:04:35 +0000: Querying all objects in the system from 10.109.10.155 ... 2017-09-30 13:04:36 +0000: Got all the info, computing table ... +-----------+-----------------+---------------+ | VM/Object | Syncing objects | Bytes to sync | +-----------+-----------------+---------------+ +-----------+-----------------+---------------+ | Total | 0 | 0.00 GB | +-----------+-----------------+---------------+\nHow to log into RVC on a Windows vCenter server ? Step1 : To launch RVC on a windows vCenter server , you will need to launch the RVC batch file as an administrator which is available under directory : “C:\\Program Files\\VMware\\vCenter Server\\rvc” considering C:\\ is the local where vcenter was installed , you may need change to the right drive if installed on a different location .\nThe rvc.bat file is going to attempt logging into your vcenter sever by default with account “administrator@vsphere.local” , however not everyone may have access to this user account or the default vsphere.local might have been changed to a different name during installation of PSC or you may want to login with your administrator account of AD which has permission on your vcenter server .\nHence you will need to alter the batch file to use your account to do that . Hence create a duplicate copy of the default RVC command and edit the user name , replace the username with the user with which you want to login with . See below example :\nStep2 : Launch rvc batch file and key in your password for the respective account and you should be in the rvc console , you may follow the navigation patch thru rvc by using cd comamnd line get to vSAN cluster (See Setp4 under appliance above) and you should be able to RVC comamnds on Windows VC .\n","link":"https://blogs.hareeshkg.bid/post/how-to-login-into-rvc/","section":"post","tags":[],"title":"How to Login into RVC?"},{"body":"How to perform vSAN cluster maintenance activities? You may be wondering what should be the Steps to reboot vSAN hosts and pre-checks before attempting a host reboot which is participating in a vSAN cluster , here is your guide to make sure your are not affecting your production in the process of a vSAN node upgrade . It is highly recommended to have a BACKUP in place for your VMs before you attempt any maintenance activity on a production cluster .\n**Step1 : **Check for VM storage policy compliance on all the virtual machines and ensure that they are compliant with their assigned VM storage policy . Fastest way is to select all VMs ⇒ Actions ⇒ VM policies ⇒ Check VM storage policy compliance . Note** : If there were any VMs with FTT=0 (Fault to tolerate) as their policy and if this is considered as an important VM , please change the policy to Virtual SAN default Policy before attempting any Host Maintenance-Mode .\nStep2 : Check for possible inaccessible objects and VMs .Easiest way is to go the health plugin (works only if we are 6.2 and above release) to see if there were any inaccessible objects , I however recommend you to use and familiarize with RVC commands as this would be your best friend !!.Its also easy to check the same status from the RVC command line on your vcenter server where your vSAN cluster resides . Please see RVC command line guide here , .First log into RVC console (see how-to-login to RVC) , change directory to the vSAN cluster , next run vsan.check_state to look for any inaccessible objects . **Step3 : **We will also need to make sure that there are no ongoing re-sync or re-balance in the cluster with the help of RVC commands on the vCenter server . **Step4 : **Increase the clom repair delay to two hours (120 mins) ONLY when you think you will need more than one hour to complete the reboot on one of the host without rebuild kick off which will causing complete resync for the components that are currently residing the host which is about to be placed in maintenance to other hosts in the cluster, proceed to put one host in maintenance mode with ensure accessibility option from the Webclient , if the DRS is not fully automated , you will have to manually move (vMotion) the VMs to other hosts . Below are the screenshots from both 6.0 and 6.5 versions where the wizard might look slightly different while changing the value clom repair delay . Note* : This step is not applicable for a three node vSAN Cluster as vSAN cannot kick of a rebuild of the components to other host because this would violate the storage policies applied on the VMs . This is only needed to avoid unnecessary rebuild operation during the process of host reboot , we proceed to Step 4 skipping this step . Also note that if there was another node failure in the cluster during the process of rebooting a node , we may compromise Virtual machine vmdk components depending on the policy used on each components. Hence it is recommended to **Always have backups **.\n**Step5 : **Proceed to put host is in maintenance mode and later you may proceed with reboot of the host and wait for it to come back online . If there were any issues while entering maintenance mode on a host , DONOT force reboot the host , contact support to investigate this further, force reboots can cause adverse affects and issues with the cluster and VMs . Note* : You have three modes to place the hosts in maintenance Mode , choose wisely what you desire , see vSAN-Host-Maintenance-Guide for detailed explanation on all three modes of maintenance.\n**Step6 : **Post reboot you must see the host back in maintenance , you may now take the host out of maintenance , you will need to watch out for possible resync , use RVC command : ” vsan.resync_dashboard .” (see how-to-login to RVC) under cluster directory . If there is an ongoing resync , DONOT attempt maintenance on any of the other hosts , wait for the resync to complete . Please contact VMware technical support if there were anomalies around resync completion or any other issue. Step7 **: **After confirming resync completion (or 0GB to resync) , you may proceed to reboot the next hosts if at you had plans for other hosts , continue following Steps 1 to 5 in a cyclic manner one host after another . For more information regarding steps and best practices see VMware KB-here which has the procedure to power down and power-up vSAN clusters and admin guides for vSAN 6.1 , 6.2 ,6.5 and 6.6 are available here\n","link":"https://blogs.hareeshkg.bid/post/how-to-perform-vsan-cluster-maintenance-activities/","section":"post","tags":[],"title":"How to perform vSAN cluster maintenance activities"},{"body":"How to Upgrade to vSAN 6.6 ? How to Upgrade vSAN cluster through Update manager? There is significant improvement when it comes to performance and the way things are handled in vSAN 6.6 , you are allowed to perform upgrades for the hosts using Update Manager available on the latest vCenter server 6.5 update 1 as VUM is now aware of vSAN enabled cluster . For all environments which are 6.0 U3 for both Vcenter server and ESXi and you are looking forward to leverage all features of VSAN 6.6 , you may now upgrade the setup to 6.5 U1 for both vCenter server and hosts . There are more information that you need to know about vSAN-6.6 I encourage you to see vsan6.6-official page , also see vSAN6.6-Performance Improvements .\nLinks : vCenter Server 6.5 u1 release notes: https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-vcenter-server-651-release-notes.html\nvCenter Server 6.5 u1 downloads: https://my.vmware.com/web/vmware/details?downloadGroup=VC65U1\u0026amp;productId=614\u0026amp;rPId=17343\nESXi 6.5 release notes: https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vsphere-esxi-651-release-notes.html\nESXi 6.5 download: https://my.vmware.com/web/vmware/details?downloadGroup=ESXI65U1\u0026amp;productId=614\u0026amp;rPId=17342\nvSAN 6.6.1 release notes: https://docs.vmware.com/en/VMware-vSphere/6.5/rn/vmware-vsan-661-release-notes.html\nWhats New with vSAN 6.6.1 vSAN is now integrated with vSphere Update Manager.\nCloud Performance Analytics for vSAN .\nVC Recovery on vSAN Datastore\nPerformance improvements when compared to previous releases (6.2 and 6.1)\nPlease also see the vSAN-upgrade best practices before you start upgrading your hosts to 6.5 Update 1 . You may follow the below procedure to upgrade the hosts post having the vcenter server upgrade to the latest 6.5 U1 .\nHow to Upgrade vSAN cluster through Update manager to 6.6? Step1 : Verify if the the vCenter server is upgraded to the correct build . Step2 : Check your update Manager tab Here you must see that update manager is able to pick up your vSAN enabled clusters available in the vCenter server and also recommends you to have the hosts patched to 6.5 U1 release to make them compliant .\nStep3 : Attach the baseline and stage the Upgrade to 6.5 Update 1 Step4 : Remediate your vSAN Cluster Note: Before you start remediating the cluster , make your you have performed pre-upgrade checks . Also ensure not to choose multiple host remediation at the same time , it is advised to contact support if maintenance mode fails on some or all hosts. It is okay to uncheck “Retry entering maintenance mode in case of a failure “. Please address all recommendations suggestion the Pre-Check Remediation .\nStep5 : Check Update manager and health Plugin tab Verify if the cluster is compliant on the update manager tab and also if there were any warnings reported on the health check plugin post upgrading all hosts .\nIn my case the health plugin was reporting about the controller firmware version detection , hence I had to push the avago vibs to the hosts to detect the firmware on the I/O controller.\n","link":"https://blogs.hareeshkg.bid/post/upgrade-to-vsan-66/","section":"post","tags":[],"title":"Upgrade to vSAN 6.6"},{"body":"Here are the troubleshooting Steps to resolve Unicastagent issues on vSAN cluster post upgrade usually from vSAN 6.1 / 6.2 / 6.5 to vSAN to 6.6 6.50d and below as we no longer use multicast from version 6.7 and unicastagent list is not updated with correct details on one or more hosts and you also vSAN network partition on one or more hosts .\nIn these cases you will need to manually add the unicastagent address list on all hosts part of the cluster , follow thru the steps listed below .\nNote :** From ESXI 6.5 Update 1 onwards , all the unicatagent entries is controlled from vCenter server and vCenter will push all the unicastagent entries to the hosts , no manual task is needed to add the unicast entries to the hosts . In very rare cases when we perform host removal and addition task we see that host will be in a network partition state and we will need to resolve this manually .\nHow and where do you Start to troubleshoot Cluster partition issue ? It is very obvious that you will see a warning on the cluster and the vSAN health plugin as seen below examples . You should be able to easily find out host(s) where are network partitioned . In the below example we see that the host1- 10.109.10.155 which is part of a 3 Node-All-Flash vSAN 6.7 cluster is network partition and we also see that there are multiple virtual machine which have gone inaccessible due to this issue .\n**Note : We should always work towards bring up the virtual machines first as they may be impacting the production , later worry about fixing the network partition , however sometimes whenever the objects were non-complaint to the VM-Storage-policy the VM objects will actually go inaccessible as the host(s) which got partition contains a component which is probably the only good copy or the latest updated component for the VM . In such case we need to bring back the isolated/partitioned host back in cluster to get the VMs accessible .\nStep1 : Go to the vSAN Health plugin (Cluster ⇒ Monitor ⇒ vSAN ⇒ Health) we see there are warnings under “Network ⇒ vSAN cluster partition” where hosts 10.109.10.156 and 10.109.10.157 are in partition 1 and Host 10.109.10.155 is in partition 2 .\nStep2 : Find out how many objects are inaccessible under the health plugin section which should throw a warning under Data ⇒vSAN object health , here in this case we see about 20 objects which have gone inaccessible . multiple other objects in reduced availability .\nStep3: Try and fix some of the inaccessible VMs by refreshing the objects status (un-register/re-register) process . If the vCenter serve which is managing this cluster is accessible it can be easily accomplished by running some RVC commands to see how many inaccessible VMs can be brought accessible .\nSSH to the vCenter server , login to the RVC console and navigate all the to the cluster . see RVC section for assistance with logging into RVC. Run Command “vsan.check_state .” (see example output below) /localhost/6.7_DC/computers/vSAN-6.7-AF3Node\u0026gt; vsan.check_state . 2018-05-02 04:06:35 +0000: Step 1: Check for inaccessible vSAN objects Detected 18 objects to be inaccessible Detected 07484b5a-490f-0505-a4c3-ecf4bbec65d8 on 10.109.10.156 to be inaccessible Detected 139f475a-5988-2a0d-4aff-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected 3794b35a-7742-a83d-d5c2-ecf4bbec65d8 on 10.109.10.156 to be inaccessible Detected 3db8b359-50f8-2d4a-3fdb-ecf4bbec6050 on 10.109.10.156 to be inaccessible Detected f746475a-ba30-8055-71d1-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected 6c44a35a-e2be-c667-c6cc-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected a630ab5a-9df8-3273-e07f-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected 109f475a-03aa-238c-e89b-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected aac9385a-698f-0097-df6a-ecf4bbec65d8 on 10.109.10.156 to be inaccessible Detected 6c44a35a-8116-3ca7-eaba-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected 9aec485a-01cf-5eb5-1f42-ecf4bbec6050 on 10.109.10.156 to be inaccessible Detected 51c2c059-55c4-87ba-172a-ecf4bbec6050 on 10.109.10.156 to be inaccessible Detected 359eb35a-f9f2-bdbd-7eb5-ecf4bbec65d8 on 10.109.10.156 to be inaccessible Detected a830ab5a-5a2f-24cc-b351-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected 3694b35a-f9c5-4dd5-6c6b-ecf4bbec65d8 on 10.109.10.156 to be inaccessible Detected 9cec485a-aee5-29e3-dff3-ecf4bbec6050 on 10.109.10.156 to be inaccessible Detected 129f475a-2ac2-edea-3e00-ecf4bbec91a8 on 10.109.10.156 to be inaccessible Detected 3694b35a-8229-9af3-325e-ecf4bbec65d8 on 10.109.10.156 to be inaccessible\n2018-05-02 04:06:35 +0000: Step 2: Check for invalid/inaccessible VMs Detected VM 'CLONED' as being 'inaccessible' Detected VM '%2fvmfs%2fvolumes%2fvsan:523d5e5605a4d751-0c3304ae7a42599b%2f04484b5a-e35b-84dc-' as being 'inaccessible' Detected VM '%2fvmfs%2fvolumes%2fvsan:523d5e5605a4d751-0c3304ae7a42599b%2fa630ab5a-9df8-3273-' as being 'inaccessible' Detected VM 'server2012' as being 'inaccessible' Detected VM 'ComposerServer' as being 'inaccessible' Detected VM 'VMware-vR-Appliance-7.3.0' as being 'inaccessible'\nNext check how many Objects/VMs can be brought online by running RVC command : “vsan.check_state . -r” (Recommended to read how to login to RVC) , in the above example we see that there are 6 VMs which were inaccessible and when this command is run you would be asked type [Y/N] as re-registering VM will cause loss of some of the management state of this VM (for eg. storage policy, permissions, tags, scheduled tasks, etc. but NO data loss) , hit Y for all the prompts and finally we should see how many of them are still inaccessible after refreshing all vms . We still see 2 VMs inaccessible which cannot be brought online by mere re-registration , which means that we don’t have 50% availability of components to keep the virtual machine objects accessible . 1/localhost/6.7_DC/computers/vSAN-6.7-AF3Node\u0026gt; vsan.check_state . -r 22018-05-02 04:07:28 +0000: Step 1: Check for inaccessible vSAN objects 3Detected 07484b5a-490f-0505-a4c3-ecf4bbec65d8 to be inaccessible, refreshing state 4Detected 139f475a-5988-2a0d-4aff-ecf4bbec91a8 to be inaccessible, refreshing state 5Detected 3794b35a-7742-a83d-d5c2-ecf4bbec65d8 to be inaccessible, refreshing state 6Detected 3db8b359-50f8-2d4a-3fdb-ecf4bbec6050 to be inaccessible, refreshing state 7Detected f746475a-ba30-8055-71d1-ecf4bbec91a8 to be inaccessible, refreshing state 8Detected 6c44a35a-e2be-c667-c6cc-ecf4bbec91a8 to be inaccessible, refreshing state 9Detected a630ab5a-9df8-3273-e07f-ecf4bbec91a8 to be inaccessible, refreshing state 10Detected 109f475a-03aa-238c-e89b-ecf4bbec91a8 to be inaccessible, refreshing state 11Detected aac9385a-698f-0097-df6a-ecf4bbec65d8 to be inaccessible, refreshing state 12Detected 6c44a35a-8116-3ca7-eaba-ecf4bbec91a8 to be inaccessible, refreshing state 13Detected 9aec485a-01cf-5eb5-1f42-ecf4bbec6050 to be inaccessible, refreshing state 14Detected 51c2c059-55c4-87ba-172a-ecf4bbec6050 to be inaccessible, refreshing state 15Detected 359eb35a-f9f2-bdbd-7eb5-ecf4bbec65d8 to be inaccessible, refreshing state 16Detected a830ab5a-5a2f-24cc-b351-ecf4bbec91a8 to be inaccessible, refreshing state 17Detected 3694b35a-f9c5-4dd5-6c6b-ecf4bbec65d8 to be inaccessible, refreshing state 18Detected 9cec485a-aee5-29e3-dff3-ecf4bbec6050 to be inaccessible, refreshing state 19Detected 129f475a-2ac2-edea-3e00-ecf4bbec91a8 to be inaccessible, refreshing state 20Detected 3694b35a-8229-9af3-325e-ecf4bbec65d8 to be inaccessible, refreshing state 21 222018-05-02 04:07:33 +0000: Step 1b: Check for inaccessible vSAN objects, again 23Detected 07484b5a-490f-0505-a4c3-ecf4bbec65d8 is still inaccessible 24Detected 139f475a-5988-2a0d-4aff-ecf4bbec91a8 is still inaccessible 25Detected 3794b35a-7742-a83d-d5c2-ecf4bbec65d8 is still inaccessible 26Detected 3db8b359-50f8-2d4a-3fdb-ecf4bbec6050 is still inaccessible 27Detected f746475a-ba30-8055-71d1-ecf4bbec91a8 is still inaccessible 28Detected 6c44a35a-e2be-c667-c6cc-ecf4bbec91a8 is still inaccessible 29Detected a630ab5a-9df8-3273-e07f-ecf4bbec91a8 is still inaccessible 30Detected 109f475a-03aa-238c-e89b-ecf4bbec91a8 is still inaccessible 31Detected aac9385a-698f-0097-df6a-ecf4bbec65d8 is still inaccessible 32Detected 6c44a35a-8116-3ca7-eaba-ecf4bbec91a8 is still inaccessible 33Detected 9aec485a-01cf-5eb5-1f42-ecf4bbec6050 is still inaccessible 34Detected 51c2c059-55c4-87ba-172a-ecf4bbec6050 is still inaccessible 35Detected 359eb35a-f9f2-bdbd-7eb5-ecf4bbec65d8 is still inaccessible 36Detected a830ab5a-5a2f-24cc-b351-ecf4bbec91a8 is still inaccessible 37Detected 3694b35a-f9c5-4dd5-6c6b-ecf4bbec65d8 is still inaccessible 38Detected 9cec485a-aee5-29e3-dff3-ecf4bbec6050 is still inaccessible 39Detected 129f475a-2ac2-edea-3e00-ecf4bbec91a8 is still inaccessible 40Detected 3694b35a-8229-9af3-325e-ecf4bbec65d8 is still inaccessible 41 422018-05-02 04:07:33 +0000: Step 2: Check for invalid/inaccessible VMs 43Detected VM \u0026#39;CLONED\u0026#39; as being \u0026#39;inaccessible\u0026#39;, reloading ... 44RbVmomi::Fault: SystemError: A general system error occurred: Invalid fault 45You have chosen to fix these VMs. 46This involves re-registering the VM which will cause loss of some of the management state of this VM 47(for eg. storage policy, permissions, tags, scheduled tasks, etc. but NO data loss). Do you want to continue [Y/N] ? 48y 49Attempting to fix the vm... 50Unregistering VM CLONED 51Registering VM CLONED 52RegisterVM Discovered virtual machine: success 53Detected VM \u0026#39;%2fvmfs%2fvolumes%2fvsan:523d5e5605a4d751-0c3304ae7a42599b%2f04484b5a-e35b-84dc-\u0026#39; as being \u0026#39;inaccessible\u0026#39;, reloading ... 54RbVmomi::Fault: SystemError: A general system error occurred: Invalid fault 55You have chosen to fix these VMs. 56This involves re-registering the VM which will cause loss of some of the management state of this VM 57(for eg. storage policy, permissions, tags, scheduled tasks, etc. but NO data loss). Do you want to continue [Y/N] ? 58y 59Attempting to fix the vm... 60Unregistering VM %2fvmfs%2fvolumes%2fvsan:523d5e5605a4d751-0c3304ae7a42599b%2f04484b5a-e35b-84dc- 61Registering VM %2fvmfs%2fvolumes%2fvsan:523d5e5605a4d751-0c3304ae7a42599b%2f04484b5a-e35b-84dc- 62RegisterVM Discovered virtual machine: InvalidArgument: A specified parameter was not correct: path 63Detected VM \u0026#39;%2fvmfs%2fvolumes%2fvsan:523d5e5605a4d751-0c3304ae7a42599b%2fa630ab5a-9df8-3273-\u0026#39; as being \u0026#39;inaccessible\u0026#39;, reloading ... 64RbVmomi::Fault: SystemError: A general system error occurred: Invalid fault 65You have chosen to fix these VMs. 66This involves re-registering the VM which will cause loss of some of the management state of this VM 67 (for eg. storage policy, permissions, tags, scheduled tasks, etc. but NO data loss). Do you want to continue [Y/N] ? 68. 69. 702018-05-02 04:16:53 +0000: Step 2: Check for invalid/inaccessible VMs 1Detected VM \u0026#39;CLONED\u0026#39; as being \u0026#39;inaccessible\u0026#39; 2Detected VM \u0026#39;server2012\u0026#39; as being \u0026#39;inaccessible\u0026#39; 3 42018-05-02 04:16:53 +0000: Step 3: Check for VMs for which VC/hostd/vmx are out of sync 5Did not find VMs for which VC/hostd/vmx are out of sync Step4: We were successful in getting at least some of the inaccessible virtual machine back online , however we still need to fix the other inaccessible VMs . As explained earlier we now need to investigate the actual hosts to find out what caused the cluster/network partition . We need check cluster members , vmkernel used for vSAN and the unicastagent entries available on all the three hosts to figure out what entry is missing in the unicast address list which caused the cluster partition over SSH to all hosts in the cluster . Run the commands “esxcli vsan cluster get” , “esxcfg-vmknic -l” and “esxcli vsan cluster unicastagent list” on all the hosts and see what entry is missing on each host . In this example we see that the host-156 and host-157 have the vSAN vmkernal unicast neighbor IP address listed of each other and is missing the entry for host-155 . However the host-155 has the entry for vSAN vmkernel IP adress in the unicast neighbhor address list for both hosts-156 and 157 . [root@is-tse-d155:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2018-05-02T03:25:38Z Local Node UUID: 5938de9a-e35b-d745-c9ff-ecf4bbec65d8 Local Node Type: NORMAL Local Node State: MASTER Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5938de9a-e35b-d745-c9ff-ecf4bbec65d8 Sub-Cluster Backup UUID: Sub-Cluster UUID: 523d5e56-05a4-d751-0c33-04ae7a42599b Sub-Cluster Membership Entry Revision: 5 Sub-Cluster Member Count: 1 Sub-Cluster Member UUIDs: 5938de9a-e35b-d745-c9ff-ecf4bbec65d8 Sub-Cluster Membership UUID: e826e95a-afff-8356-8c7c-ecf4bbec65d8 Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: e1acbef4-fc1e-4901-b365-091166f8d30e 4 2017-09-16T19:31:38.194\n[root@is-tse-d155:~] esxcfg-vmknic -l Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk1 vSAN-network IPv4 10.109.44.30 255.255.240.0 10.109.47.255 00:50:56:68:12:89 1500 65535 true STATIC defaultTcpipStack\n[root@is-tse-d155:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name\n5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 0 true 10.109.44.31 12321 5937c679-f343-be43-49a3-ecf4bbec6050 0 true 10.109.44.32 12321\n[root@is-tse-d157:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2018-05-02T03:25:24Z Local Node UUID: 5937c679-f343-be43-49a3-ecf4bbec6050 Local Node Type: NORMAL Local Node State: BACKUP Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 Sub-Cluster Backup UUID: 5937c679-f343-be43-49a3-ecf4bbec6050 Sub-Cluster UUID: 523d5e56-05a4-d751-0c33-04ae7a42599b Sub-Cluster Membership Entry Revision: 0 Sub-Cluster Member Count: 2 Sub-Cluster Member UUIDs: 5937c679-f343-be43-49a3-ecf4bbec6050, 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 Sub-Cluster Membership UUID: 6b83e15a-2033-2928-572e-ecf4bbec91a8 Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: e1acbef4-fc1e-4901-b365-091166f8d30e 11 2018-05-02T02:48:04.503\n[root@is-tse-d157:~] esxcfg-vmknic -l Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack vmk1 vSAN-network IPv4 10.109.44.32 255.255.240.0 10.109.47.255 00:50:56:65:5b:ea 1500 65535 true STATIC defaultTcpipStack\n[root@is-tse-d157:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name\n5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 0 true 10.109.44.31 12321\n[root@is-tse-d156:~] esxcli vsan cluster get Cluster Information Enabled: true Current Local Time: 2018-05-02T04:03:29Z Local Node UUID: 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 Local Node Type: NORMAL Local Node State: MASTER Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 Sub-Cluster Backup UUID: 5937c679-f343-be43-49a3-ecf4bbec6050 Sub-Cluster UUID: 523d5e56-05a4-d751-0c33-04ae7a42599b Sub-Cluster Membership Entry Revision: 0 Sub-Cluster Member Count: 2 Sub-Cluster Member UUIDs: 5937c679-f343-be43-49a3-ecf4bbec6050, 5937c663-8cb8-3d48-d3ad-ecf4bbec91a8 Sub-Cluster Membership UUID: 6b83e15a-2033-2928-572e-ecf4bbec91a8 Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: e1acbef4-fc1e-4901-b365-091166f8d30e 11 2018-05-02T02:48:04.441\n[root@is-tse-d156:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name\n5937c679-f343-be43-49a3-ecf4bbec6050 0 true 10.109.44.32 12321 [root@is-tse-d156:~] esxcfg-vmknic -l Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type vmk1 vSAN-network IPv4 10.109.44.31 255.255.240.0 10.109.47.255 00:50:56:6e:0f:a4 1500 65535 true STATIC defaultTcpipStack\n[root@is-tse-d156:~] esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name\n5937c679-f343-be43-49a3-ecf4bbec6050 0 true 10.109.44.32 12321\nStep5: In vSAN 6.6 and above the we have a new check/test added into the vCenter server under vSAN health plugin called “vCenter state is authoritative” under vSAN-health plugin (Cluster⇒ vCenter state is authoritative ) which checks and manages the unicast entries on all the hosts along with few other tasks . In this case we see that there is a warning on this check reporting that the last update on this cluster was by a different Vcenter server which may the actual cause of the problem . This is possible if the cluster was moved from a different vcenter server to this vCenter and then hit the network partition issue . We may not need to add the unicast entries manually to fix the network partition instead hit “Update ESXi Configuration” this would fix the issue . In this scenario I had removed the hosts from a different vcenter server and added them to this new cluster and hit this problem . Later I had found out that there was a setting mismatch on host-155 where I had set the value as 1 for “/VSAN/IgnoreClusterMemberListUpdates” previously which didnt allow the vcenter server to autmatically add/fix unicast issues . setting this value back to zero and clicking “Update ESXi Configuration” helped fix the problem . [ Image not available: Update-ESXi-configuration.png\n[root@is-tse-d155:] esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates Value of IgnoreClusterMemberListUpdates is 1 [root@is-tse-d155:] esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates Value of IgnoreClusterMemberListUpdates is 0\nIf the above step didnt help fix the problem , make sure command \u0026quot;esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates\u0026quot; is run on all the hosts and they retry \u0026quot;Update ESXi Configuration\u0026quot; otherwise unicast entries might have to be manually added to all the hosts . Reach out to VMware support to help/assist you add the unicast values on each host .\n**Note* : Please “DO NOT” attempt adding unicast entries on your own ,adding wrong entry in the Unicast entry causes hosts to PSOD . Please Engage VMware vSAN support to further help fix your issue .** Step6 : After Network apparition issue is fixed , there will be a slight resync for the objects which are not up to date once all the resync finishes , run Retest on the vSAN-health plugin , all health check test should come clean .\n/localhost/6.7_DC/computers/vSAN-6.7-AF3Node\u0026gt; vsan.resync_dashboard . 2018-05-02 05:08:41 +0000: Querying all VMs on vSAN ... 2018-05-02 05:08:41 +0000: Querying all objects in the system from 10.109.10.156 ... 2018-05-02 05:08:41 +0000: Got all the info, computing table ... +--------------------------------------------------------------------------------+-----------------+---------------+ | VM/Object | Syncing objects | Bytes to sync | +--------------------------------------------------------------------------------+-----------------+---------------+ | 3Node-AF-VCVA67 | 2 | | | [vsanDatastore] 5728e95a-29b0-552d-5860-ecf4bbec91a8/3Node-AF-VCVA67_1.vmdk | | 1.84 GB | | [vsanDatastore] 5728e95a-29b0-552d-5860-ecf4bbec91a8/3Node-AF-VCVA67_6.vmdk | | 0.51 GB | | 3Node-AF-PSC67 | 1 | | | [vsanDatastore] a427e95a-389e-a03d-e009-ecf4bbec91a8/3Node-AF-PSC67.vmdk | | 2.28 GB | +--------------------------------------------------------------------------------+-----------------+---------------+ | Total | 3 | 4.63 GB | +--------------------------------------------------------------------------------+-----------------+---------------+\nRetest : [ ![](/images/troubleshooting-vsan-cluster-partition-vsan-67-and-vsan-66-Object-Status-after-nw-paritionFix-1024x370.png) ](/images/troubleshooting-vsan-cluster-partition-vsan-67-and-vsan-66-Object-Status-after-nw-paritionFix.png) [ ![](/images/troubleshooting-vsan-cluster-partition-vsan-67-and-vsan-66-Network-PartitionFix-1024x394.png) ](/images/troubleshooting-vsan-cluster-partition-vsan-67-and-vsan-66-Network-PartitionFix.png) ","link":"https://blogs.hareeshkg.bid/post/troubleshooting-vsan-cluster-partition-vsan-67-and-vsan-66/","section":"post","tags":[],"title":"Troubleshooting vSAN cluster partition vSAN 6.7 and vSAN 6.6"},{"body":"Here is your simple “How to Upgrade vSAN with best practice guide ?” . I have been asked by a lot of people recently regarding vSAN upgrades sequence , questions will generally be like ,what are my threats? , what are the prerequisites ? , Should I perform upgrades on all hosts together or one node after another ? what are my checks before I proceed to upgrade vSAN hosts? what are the dependencies before I start upgrading my hosts .However you may ask why should I upgrade my hosts / VCenter servers ? The answer to that is very simple , based on certain known issues seen on each version or builds the fixes are provided in the next patches /updates or version releases . It is recommended to get the hosts to latest patches available at the current version if not going forward with the next version release .\nIf you are running vSAN 6.1 / 6.2 with hosts on 6.0 U1 / 6.0 U2 , it is highly recommended to get to the latest build “**ESXi 6.0 U3 **see **Link **and Link“ where multiple known issues with vSAN have been addressed , also considering vSAN 6.6.1 is always a better option . Similarly if you are running a vSAN 6.5 (ESXi 6.5 GA or 6.5 a release) it is recommended to get to the latest build of “ESXi 6.5 U1 see Link and Link” which is vSAN 6.6.1.\nBefore you start to think about upgrades , we need to make a list of things to be addressed along with your host upgrades , it may be your host BIOS , firmware , I/O Controller firmware and drivers , network card firmware and drivers . This will save you significant time and avoid requirement of multiple reboot if they were put together smartly . I would also high encourage you to go thru VMware official Upgrade Docs here where few additional steps are explained .\n**Upgrade recipe **Pre-Checks Check for possible inaccessible objects and VMs in the cluster and log a case with VMware support if you could not find out what those objects were .\nVirtual machine compliance status check.If there were any VMs with Fault-to-Tolerate (FTT) = 0 , do you need them if yes , consider changing the policy to default or any other data redundant policy .\nCheck Current utilization of vSAN datastore , Check What would be your Possible Resync/Rebuild data when a host goes down and can I accommodate the data on rest of the host when one host goes down. Also ensure that there are No Current Ongoing Resync before attempting any maintenance mode mask on the cluster .\nCheck the Utilization of each disk group and disks , check if they need a Proactive Rebalance .\nEnsure to have the Latest BACKUP for all VMs .\nCheck and Verify current Driver and firmware for I/O controller and network card , Disk Firmware (Cache and capacity) .\nEnsure that There is NO Critical Errors on the vSAN health plugin (Note* only 6.2 and above version only has a working health plugin ) , before starting the upgrade process\n**Upgrade recipe **Pre-checks Explained 1.Check for possible inaccessible objects and VMs Easiest way is to go the health plugin (works only if we are 6.2 and above release) to see if there were any inaccessible objects , I however recommend you to use and familiarize with RVC commands as this would be your best friend !!.Its also easy to check the same status from the RVC command line on your vcenter server where your vSAN cluster resides . Please see RVC command line guide here .First log into RVC console (See how to log into RVC) , change directory to the vSAN cluster , next run vsan.check_state to look for any inaccessible objects .\n2. Virtual machine compliance status check. It is very important to ensure that the VMs in the vSAN datastore are complaint with their set policies always . We may miss out some important things and proceeding with upgrades of hosts might have adverse affect if the VMs which were non complaint we may end up in a possible data loss situation . Also if there were VMs which were using FTT=0 policy you may either convert them to vSAN default policy and hit apply if they were required or you can have those VMs deleted to avoid data loss situation . If you find any VMs that were non complaint to their policy however if the objects /components were shown active you may simply check for compliance they should turn green by right click ⇒ VM policies ⇒ check for compliance . If this did not work simply choose re-apply policy and they should go complaint .\n3. Check Current utilization of vSAN datastore , Check What if host failure scenario Ensure that we have enough datastore free space on the vSAN datastore to accommodate data from complete host , since performing upgrades per host basis may trigger a complete resync/rebuild of the host that is under maintenance .Again here RVC is your best friend , run “vsan.whatif_host_failures .” to see vSAN datastore utilization after a host is taken down , also if there were any ongoing resync .\n4.Check the utilization of each disk group and disks , check if they need a proactive rebalance . It is good to check if there were any over utilized disk groups or disks before starting an upgrade procedure . RVC command ” vsan.disks_stats .” at the cluster level shows the disk group current utilization , also the command “vsan.proactive_rebalance_info .” , will show you how much of data needs to be re-balanced between Disk groups and disks .\n5. Ensure and take latest backups for all VMs . We all understand that the upgrades are non-disruptive but “Does it mean nothing can go wrong?” , it is always good to have backups and is a highly recommended per best practices for all VMs\n6. Check and Verify current Driver and firmware for I/O controller , Disk Firmware (Cache and capacity) and network card drivers and firmware . Please note you may not be able to get the current firmware from ESXi , you will need to use your hardware console for the hosts (Dell : iDRAC , HP : ILO , CISCO : KVM , etc) from the respective vendor to determine the firmware for your controller , nic cards and otehr peripherals .\nTo compare the current driver for compatibility you will need to check VMwareHCL and VMware-vSAN-HCL respectively for your network card /driver and vSAN compatiblity for I/O Controller compatiblity with vSAN version / Host version , recommended driver/firwmare release , SSD / HDD compatibility and recommended version .\nHere is an example to find the current controller driver and firmware and compare it with VMware certified driver and firmware, if you found that there was a mismatch , please proceed to correct them per recommended version on the respective HCL guide . You will need to follow similar steps to identify the nic cards in use on the host and their current drivers and firmware , also have them upgraded per HCL guide .Please follow KB Article see verify your current driver and firmware for all nic cards and HBA .\nStep1 : Find the current I/O controller in use for vSAN [root@is-tse-d155:~] esxcfg-scsidevs -a vmhba0 lsi_mr3 link-n/a sas.514187706c076000 (0000:02:00.0) Avago (LSI) Dell PERC H730 Mini vmhba1 vmw_ahci link-n/a sata.vmhba1 (0000:00:11.4) Intel Corporation Wellsburg AHCI Controller vmhba2 vmw_ahci link-n/a sata.vmhba2 (0000:00:1f.2) Intel Corporation Wellsburg AHCI Controller vmhba64 iscsi_vmk online iscsi.vmhba64 iSCSI Software Adapter vmhba32 vmkusb link-n/a usb.vmhba32 () USB\nStep2 : Find the current Driver in use for the controller 1[root@is-tse-d155:~] vmkload_mod -s lsi_mr3 | grep Version 2 Version: 6.910.18.00-1vmw.650.0.0.4564106 Step3 : Get the VID DID SVID SSID for search your component accurately on the HCL guide 1[root@is-tse-d155:~] vmkchdev -l | grep vmhba0 20000:02:00.0 1000:005d 1028:1f49 vmkernel vmhba0 Step4 : Verify the hardware against the VMware vSAN HCL guide I would prefer the website : [https://hcl.captain-vsan.com](https://hcl.captain-vsan.com) where we can enter the above details which will direct you to the [VMware-vSAN-HCL](https://www.vmware.com/resources/compatibility/search.php?deviceCategory=vsan) post you fill in the details for : VID DID SVID SSID , please find the details below . From this step you will now be aware of the recommended driver and firmware combination which is certified for vSAN for the respective controller . Step5 : Verify the current Firmware against the drives in use for vSAN , if they also need any upgrade Please do note that firmware revision on drives (both Cache and capacity) should be at par or higher as described on the vSAN-HCL guide and is very critical , as we issues related to race conditions between Controller and disks causing all disks in the host to go into permanent device loss and hosts go into non-responsive state , triggering a massive resync and also cause cluster wide issues like congestion etc..\nTo validate the current firmware you may use the command esxcli storage core device list and check the drive revision which corresponds to the firmware for the drive and check the vSAN-HCL for the recommended Firmware on the drives .\nEX :\n[root@is-tse-d155:~] esxcli storage core device list | less\nnaa.500003969809b581 Display Name: Local TOSHIBA Disk (naa.500003969809b581) Has Settable Display Name: true Size: 286102 Device Type: Direct-Access Multipath Plugin: NMP Devfs Path: /vmfs/devices/disks/naa.500003969809b581 Vendor: TOSHIBA Model: AL13SXB30EN Revision: DK02 SCSI Level: 6 Is Pseudo: false Status: on Is RDM Capable: true Is Local: true Is Removable: false Is SSD: false Is VVOL PE: false Is Offline: false Is Perennially Reserved: false Queue Full Sample Size: 0 Queue Full Threshold: 0 Thin Provisioning Status: unknown Attached Filters: VAAI Status: unsupported Other UIDs: vml.0200000000500003969809b581414c31335358 Is Shared Clusterwide: false Is Local SAS Device: true Is SAS: true Is USB: false Is Boot USB Device: false Is Boot Device: true Device Max Queue Depth: 64 No of outstanding IOs with competing worlds: 32 Drive Type: physical RAID Level: NA Number of Physical Drives: 1 Protection Enabled: false PI Activated: false PI Type: 0 PI Protection Mask: NO PROTECTION Supported Guard Types: NO GUARD SUPPORT DIX Enabled: false DIX Guard Type: NO GUARD SUPPORT Emulated DIX/DIF Enabled: false\nWith the Model Number and revision number you see against the disk from above example you may verify the recommended Firmware Revision for the disk . Follow thru the screenshots to verify the disk firmware . Now you have successfully completed the Pre-upgrade check .\n\u003e Begin Upgrade Process Lets discuss the upgrade options , you may choose to do this in two ways , each have their own benefits and cons . You may choose to proceed with either of the them depending on your requirements\nWith a full downtime on all the VMs , complete the upgrade on all hosts and then bring up the environment once you have successfully completed the upgrade .\nUpgrade with no downtime, maintenance mode with ensure accessibility\nUpgrade with no downtime, with full data migration\nRecipe 1 : Upgrade with Full Downtime This approach to upgrade is the most preferred in many cases . The steps are very straight forward , all you need is a downtime on all your Virtual machines running residing on the cluster which is about to be upgraded . Please ensure you have followed all pre-upgrade checks before proceeding with these steps\n**Step 1 : **Shutdown all the virtual machines on the cluster .\nStep 2 : Check for VM storage policy compliance on all the virtual machines and ensure that they are compliant with their assigned VM storage policy . Note : If there were any VMs with FTT=0 (Fault to tolerate) as their policy and if this is considered as a critical VM , please change the policy to Virtual SAN default Policy before attempting any Host Maintenance-Mode .\nStep 3 : Once you have confirmed all VMs are powered and and their compliance check is clean , proceed to put all hosts to MaintenanceMode with NO-DATA-MIGRATION .\n**Step 4 : **You may now start upgrade all hosts in one go to the build that is recommended / desired , perform all the required upgrades at the hardware layer (BIOS / Firmware etc) .\nStep 5 : Once all hosts are upgraded to the latest build, you may next push the driver vibs for network cards , controller driver and any additional patches for the host post upgrade and take another reboot on all hosts . Now you would have completed the upgrade sequence and you may exit Maintenance Mode on all hosts .\n**Step 6 : **VMs can now be powered on / brought back to production .\nThe advantage of this option is that there were be no resync that is required post upgrade on all hosts , hence post upgrade VMs can be brought online and complete vSAN bandwidth is available for VM-storage traffic , the only drawback is the requirement of a downtime , it may not be feasible in all production scenarios .\nRecipe 2 : Upgrade with No Downtime This approach to upgrade is preferred in situations where there can be no down time on the VMs that are running which are mostly mission critical businesses. The steps are explained below , please ensure you have followed all pre-upgrade checks before proceeding with these steps .\n**Step1 :**Check for VM storage policy compliance on all the virtual machines and ensure that they are compliant with their assigned VM storage policy . Note : If there were any VMs with FTT=0 (Fault to tolerate) as their policy and if this is considered as a critical VM , please change the policy to Virtual SAN default Policy before attempting any Host Maintenance-Mode .\n**Step2 : **Once all VMs are checked for compliance , you will also need to make sure that there are no ongoing re-sync or re-balance in the cluster with the help of RVC commands on the vCenter server .\n**Step3 : **Increase the clom repair delay to two hours (120 mins) ONLY when you think you will need more than one hour to complete the upgrade on one of the host without rebuild kick off which will causing complete resync for the components that are currently residing the host which is about to be placed in maintenance to other hosts in the cluster, proceed to put one host in maintenance mode with ensure accessibility option from the Webclient , if the DRS is not fully automated , you will have to manually move (vMotion) the VMs to other hosts . Below are the screenshots from both 6.0 and 6.5 versions where the wizard might look slightly different while changing the value clom repair delay .\n**Step4 : **Once the first host is in maintenance mode , you may proceed with all of the upgrades . Start with the host upgrade to the latest build version , proceed with any recommendation on the hardware side for BIOS / Firmware etc (this includes for your I/O controller firmware , SSD/HDD firmware , nic firmware) . Once these upgrades are completed , you may now check if the hosts require any additional patches , driver upgrades for your I/O controller , nics . Push them all in one go and take another reboot . This host can now exit maintenance post all the upgrades and patches .\n**Step5 : **Post taking the first host out of maintenance , you will need to watch out for possible resync , use RVC command : ” vsan.resync_dashboard .” under cluster directory . If there is an ongoing resync , DONOT attempt maintenance on any of the other hosts , wait for the resync to complete . Please contact VMware technical support if there were anomalies around resync completion or any other issue.\nStep6 **: **After confirming resync completion (or 0GB to resync) , you may proceed to upgrade the next host following Steps 1 to 5 in a cyclic manner one host after another .\nRecipe 3 : Upgrade with No Downtime and full data migration. This approach to upgrade is preferred in situations company policies which dictates no downtime , data compromise . This is only applicable for more than three node vSAN normal cluster and if in fault domains we should have enough free space on the other hosts within the same fault domain to achieve this.This process is time consuming and will take longer time to complete the upgrade on all hosts as there are multiple resync cycles involved .Please ensure you have followed all pre-upgrade checks before proceeding with these steps\n**Step1 :**Check for VM storage policy compliance on all the virtual machines and ensure that they are compliant with their assigned VM storage policy . Note : If there were any VMs with FTT=0 (Fault to tolerate) as their policy and if this is considered as a critical VM , please change the policy to Virtual SAN default Policy before attempting any Host Maintenance-Mode .\n**Step2 : **Once all VMs are checked for compliance , you will also need to make sure that there are no ongoing re-sync or re-balance in the cluster with the help of RVC commands on the vCenter server .\n**Step3 : **Proceed to put one host in maintenance mode with full data migration , wait for the full resync completion . You may monitor resync using RVC , host will only complete entering maintenance mode post reysnc completion .\n**Step4 :**Once the first host is in maintenance mode , you may proceed with all of the upgrades . Start with the host upgrade to the latest build version , proceed with any recommendation on the hardware side for BIOS / Firmware etc (this includes for your I/O controller firmware , SSD/HDD firmware , nic firmware) . Once these upgrades are completed , you may now check if the hosts require any additional patches , driver upgrades for your I/O controller , nics . Push them all in one go and take another reboot . This host can now exit maintenance post all the upgrades and patches .\n**Step5 : **Post taking the first host out of maintenance , you will need to watch out for possible resync , use RVC command : ” vsan.resync_dashboard .” under cluster directory . If there is an ongoing resync , DONOT attempt maintenance on any of the other hosts , wait for the resync to complete .\n** Step6 : **After confirming resync completion (or 0GB to resync) , you may proceed to upgrade the next host following Steps 1 to 5 in a cyclic manner one host after another .\nNote* : There are chances that the vSAN cluster gets network partitioned post upgrade especially from vSAN 6.1 / 6.2 / 6.5 to vSAN to 6.6 as we no longer use multicast from version 6.6 . In these cases you will need to manually add the unicastagent address list on all hosts part of the cluster , I encourage you to see Troubleshooting-Unicastagent\nUpgrading vSAN disk Format version Disk format upgrade may either from the Vcenter webclient or thru RVC , I suggest you first meet the pre-requisite for Disk format upgrade prior to attempting this . I personally prefer RVC to do this , recommend you to refer to the links RVC-Method or Webclient-method . Please also refer to Link for better understanding “on Disk format upgrades and** Pre-Requisites” **. Refer to How to log into RVC\n[contact-form][contact-field label=’Name’ type=’name’ required=’1’/][contact-field label=’Email’ type=’email’ required=’1’/][contact-field label=’Website’ type=’url’/][contact-field label=’Comment’ type=’textarea’ required=’1’/][/contact-form]\n","link":"https://blogs.hareeshkg.bid/post/how-to-upgrade-vsan-cluster-with-best-practices/","section":"post","tags":[],"title":"How to Upgrade vSAN CLuster with Best practices"},{"body":"So far you must have seen , how to deploy and complete UnityVSA initial configuration here . Now lets see How to configure Pool, Luns and SRA for SRM -UnityVSA .This is a perfect and easy solution to configure Array based replication LAB for SRM .\nLets get Started !!\nStep 1 : Add a new virtual disk to Unity-VSA from vSphere Right click your UnityVSA virtual machine , allocate a new virtual disk to be used for pool creation . Please note for the free Edition the disk size can only go up to 4 TB . You will need to do the same process at both sites .\nStep 2 : Create a new Pool from the Unisphere and complete the Pool creation wizard . Follow the same process at both primary and the DR site .\nStep 3 : Add the vcenter or individual hosts to ACCESS List Step 4 : Create the iSCSI interface if not created during the initial configuration wizard Once you have configured the pool and iSCSI interface you can head to initiators under access , there you will not find any initiators from your hosts which are part of the vcenter logged in , for this to happen you will need to crate/configure your software iSCSI adapter on all hosts that should have access to your UNITY-VSA storage . Lets see how its done .\nStep5 : Configure software-iSCSI adapter on host to have access to your UnityVSA storage Perform this in all hosts which should have access to the Luns.\nStep 6 : Create, export Luns and configure datastore To create a Lun , we need to first make sure that the Pool creation was successful . Once we have the pool ready , we can create N number of Luns whose total size should be less than or equal to the size of the Pool .In our case we have a 100GB pool , out of which we have created two Luns of size 50GB and 5gb.\nStep 7 : Create a datastore using the Luns exported from UnityVSA Step 8 : Configure Lun replication between UnityVSA primary site and DR site **Pre-requisite : **Make sure you deploy another UnityVSA at your DR site , complete initial configuration wizard , create a pool , configure iSCSI interface , add DR vcenter hosts to the access list . Configure software-iSCSI adapter on the host at DR site and add dynamic discovery with the iSCSI interface IP created at the UnityVSA DR site .\nConfigure Replication interface at Primary and DR site, Unisphere⇒ DataProtection ⇒ Replication ⇒ Interface . Ensure status goes to enabled .\nCreate Replication connection between two UnityVSA , Unisphere⇒ DataProtection ⇒ Replication ⇒ Connections . Here we would be asked to enter the mangement IP address of the DR UnityVSA ,username/password and click OK .Ensure connection status is also OK .\nNow configure replication for the Luns created at the PR-Site . Unisphere ⇒ Storage ⇒ Block ⇒ Luns ⇒ Select Lun ⇒ Edit Properties ⇒ Replication ⇒ Configure Replication , complete the wizard .\nStep 9 : Download and install the SRA adapter Download the SRA for Unity-VSA (EMC Unity Block Storage Replication Adapter) here .\nInstall the UnityVSA SRA adapter at each site on the respective SRM boxes at both sites .\nVerify if we can see the SRA adapter on SRM plugin for both sites .\nAdd array manager for both sites and enable array pairs . Ensure you see the array pair status OK and also see the direction of replication for the Luns correctly between sites .\nYou have now completed Array based replication LAB for SRM with help of EMC UnityVSA .\n","link":"https://blogs.hareeshkg.bid/post/how-to-configure-pool-luns-replication-and-sra-for-srm-unityvsa/","section":"post","tags":[],"title":"How to configure Pool, Luns, replication and SRA for SRM -UnityVSA"},{"body":"Learn everything about vsan setup and administration , to know about most of things around vSAN you will need to understand very few basic things/checks on your environment which will help you monitor your environment better . The general queries that comes to everyone’s mind which I can think of are as below . For more information regarding vSAN administration and best practices also refer VMware vSAN Admin Guides which are available here\nHow to validate my Design ?\nHow can I monitor my vSAN health and what tools can I use to validate my health status?\nWhat are things I can look and share while contacting VMware?\nWhat are my best practices ?\nI am I running a good stable version of vSAN ? Should I upgrade the setup , what are my challenges in doing so?\nWhat are my precautions in need to consider before rebooting a host participating in a vSAN Cluster?\nWhat is a valid design for a vSAN HCI I would say a very first link which I would refer to storagehub-vSAN , this has most of the things that you must be looking for from a new design stand point.\nIf you are design a new vSAN hyper converged infrastructure the probable considerations for you options while procure your hardware is the VxRAIL from DellEMC which is currently having a very huge market in terms of performance , availability , scalability , manageability and customer experience . You may check out the latest features , please see the SPEC-SHEET , TECH-BOOK and** Planning Guide** . We also do have “vSAN ready nodes” which are validated with certain partners of VMware . Hence your deployment is hassle free , especially for VxRail EMC will proactively will send you the network requirements for the appliance , vcenter and ESXI hosts , if all these are kept ready your deployment would take under 30 minutes,from a support standpoint all you need to do is just open one case with EMC .\nIf you have some existing server nodes available already in the data center and you wish to convert them to a vSAN-HCI . You will need to firstly check your current hardware compatibility with vSAN . Please see the VMware-vSAN-HCL , here we have three category I/O controller , SSD and HDD . I would suggest to check out vSAN-H/W-Quick-reference guide , or manually check for compatible I/O controller , SSD and HDD for your requirement .\nA very important consideration is to have the recommended “I/O controller,SSDs and HDDs is to the Firmware and driver match per vSAN HCL” . We have seen complicated issues if the drivers and firmware does not match per HCL and hence very reason for a separate HCL guide for vSAN .\nMonitor vSAN and health from GUI What and How to monitor my vSAN health . First thing is you will need to Navigate to vSAN health service by Navigating from Cluster ⇒ Monitor ⇒ vSAN ⇒ Health . You may or may not see any warning different Metrics test generally run by the vSAN health service , here we have “Hardware Compatibility , Performance Service , Network , Physical disk , Data , Cluster , Limits and Online health” . Warnings thrown under each metrics will further needs to be examined by expanding each of them .\nHere is an example , where we see that the Hardware compatibility and performance service are having issues reported . When we expand these we see that the controller firmware is not being pulled properly by vSAN , you will definitely see a warning against vSAN HCL DB up-to-date you will need to update this either with update this by clicking get latest version online if the VC has connectivity to the internet otherwise manually upload it by following article here\nPerforming Firmware Upgrade made easy with vSAN – 6.6 You will need a vib file on each host which will poll the firmware information from the hardware and display it on the health service . This is made very simple with vSAN 6.6 , however earlier this had to be manually pushed to each host from the hardware management page HP/Dell/Lenovo..etc . On vSAN 6.6 , we can simply Navigate from Cluster ⇒ Configure ⇒ vSAN ⇒ Updates ,here we get recommendation to install the respective management tool for hardware which is installed on your hosts which will help pull the required firmware version online and also have them pushed to each host . Once the management tool is installed on all your hosts we should be able to see the current firmware on each host .You may see a warning for maintenance mode vs DRS , please choose to enable DRS on the cluster and the health plugin should automatically proceed to put hosts into maintenance mode and reboot them . However installing Mgmt-tool may not require a reboot in every case.\nOnce the required Management Tools is installed on all hosts participating in your cluster , you will now be prompted with the available latest recommended firmware for the I/O controller used for vSAN .You will need to again download from the Updates menu , then push it to the hosts , if DRS is enabled on the cluster the vSAN health service will automatically put hosts into maintenance mode update the firmware bring the hosts out of maintenance mode one by one .\nOnce all hosts are rebooted one by one you should be able to see the current firmware and recommended firmware match on the Cluster ⇒ Monitor ⇒ vSAN ⇒ Health , all warnings against configuration hardware should be cleared .\nDrivers not detected properly on vSAN Many times we see that the recommended driver on the vSAN HCL DB is already installed , however we see the current version detected as N/A . In these situations , please follow the below steps to overcome this problem .\nI have taken an example in my case , however these may differ with different hardware and recommended drivers on vSAN HCL guide .\nStep1 : Check the current version of the driver installed [root@localhost:~] esxcfg-scsidevs -a\nvmhba0 lsi_mr3 link-n/a sas.514187706c076000 (0000:02:00.0) Avago (LSI) Dell PERC PERC H710\nStep2 : Check the driver which is loaded to manage the I/O controller card : esxcli system module list | less\nHere we will find the respective driver which is in use for the host , in this case it would be lsi-mr3 and driver that is loaded and currently in use to handle your I/O controller Dell PERC PERC H710 .\nStep3 : Replace the driver with the correct one If you have already installed the latest driver as mentioned in the vSAN HCL guide Link for the above I/O controller here in this case its a **“megaraid_sas version 6.603.55.00.1vmw” , **then your job is pretty easy all we need to do is disable the driver in use and then enable the correct driver .\nNote : You will need to have the host in maintenance mode before you proceed with this action , a full data migration is not required in this case “ensure accessibility mode” should be fine .\nTake ssh to the host and execute similar command as seen below for your respective driver that needs to be disabled ,this simply disable the lsi_mr3 driver in my case , next we will need to enable the correct driver .\nTo Disable wrong driver :\nesxcli system module set –enabled=false –module=lsi_mr3\nTo Enable and load Correct Driver :\nesxcli system module set –enabled=true –module=megaraid_sas\nesxcli system module load –module=megaraid_sas\nIf at all you see that you had a lower version of the correct driver are if you wish to upgrade/re-install it before enabling it , please download the respective latest driver upload it to the host using winscp on tmp directory or a datastore , later use the esxicli software vib to re-install the correct driver and then finally reboot the host .\nExample :\nRemove Existing Driver :\n[root@localhost:/tmp] esxcli software vib remove –vibname=scsi-megaraid-sas\nRemoval Result\nMessage: The update completed successfully, but the system needs to be rebooted for the changes to be effective.\nReboot Required: true\nVIBs Installed:\nVIBs Removed: VMware_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.600.0.0.2494585\nVIBs Skipped:\nInstall Correct Driver :\n[[root@localhost:/tmp] esxcli software vib install -d “/tmp/megaraid_sas-6.606.06.00-offline_bundle-2351571.zip”\nInstallation Result\nMessage: The update completed successfully, but the system needs to be rebooted for the changes to be effective.\nReboot Required: true\nVIBs Installed: LSI_bootbank_scsi-megaraid-sas_6.606.06.00-1OEM.550.0.0.1331820\nVIBs Removed:\nVIBs Skipped\n","link":"https://blogs.hareeshkg.bid/post/vsan-deployment-and-administrators-guide/","section":"post","tags":[],"title":"vSAN deployment and administrator’s Guide"},{"body":"","link":"https://blogs.hareeshkg.bid/categories/vvol-lab/","section":"categories","tags":null,"title":"VVOL LAB"},{"body":"How to build a perfect vVOL lab setup , SRM Array based replication using UnityVSA . All you need is the DELL-EMC Unity VSA latest version , which can be downloaded here as an OVA . Once you have that downloaded , you will now proceed to deploy that on you ESXi host (5.5 and above) , please make sure that you have enough resource on the host 12GB RAM and 2 vCPU available to get this booted up successfully . This guide shows you how you can configure vVOL lab setup , SRM Array based replication UnityVSA . Please feel free to comment on this article if you could not find what you were looking for and if you have any suggestions.\nLets Get started.\nStep1 : Download the OVA from the link here , you might want to create an account if you dont have one on the their website before being able to download. Step2 : Proceed to deploy the OVA on your host which has enough resources per the requirements of the unity VSA . This deployment takes about 20 -30 minutes to complete , once you are able to see the IP address on the VM , try to open it as an URL , you must be seeing the EMC unisphere login page , you must be able to login with user **“admin” **and password “Password123#” you may choose to refer to their techincal-document or just continue with the steps here .\nStep3 : You have now successfully deployed your UnityVSA, lets continue with the initial configuration Unisphere wizard . Get to the Unisphere Licenses tab , you now will be looking at a newly generated System UUID which will automatically be generated post you deploy your appliance , you will need to login again in the EMC portal here and then paste your System UUID and generate your License and download it . You will have a **.LIC **file you will later be able to click on install license and upload your license file to unlock all features of your appliance . Proceed with the configuration of your DNS servers , NTP servers . you may skip Pools , Alert Settings ,iSCSI Interface may be configured and be tagged to one of the available ports on the VSA see the example below , you will be later using different up links for your NAS server configuration and other networking requirements for exporting your vSA storage to the hosts .You may skip the NAS server configuration part , this can be done later . You would have now completed the initial configuration for UnityVSA . To understand Pool creation , connecting hosts , creating / exporting luns and Exporting NFS storage . Please click the link here (This link is under construction**)\n","link":"https://blogs.hareeshkg.bid/post/vvol-lab-setup-srm-array-based-replication-using-unityvsa/","section":"post","tags":[],"title":"vVOL lab setup , SRM Array based replication using UnityVSA"},{"body":"I recently came across a customer who was trying to download .vmem from the vSAN datastore, so that the memory dump can be shared with the Guest OS vendor like Microsoft for core dump analysis . Since vSAN datastore is an object store file system , the OSFS namespace will only contain the pointer file of about 118 bytes which points to the associated object within vSAN datastore similar to all the VMDK files and v-swap file .\nThe following work around helps you to download the vmem file from the vsan datastore .\nOption 1 :\nIf we have a local datastore which should have equivalent to the size of the .vmem file (memory size of the virtual machine) or greater free space available then\nBrowse the datastore , select the .vmem file .\nNext drag and drop the .vmem file from the vSAN datastore to the local datastore .This will initiate a full copy of the pointer file and the associated vSAN object to the local datastore\nNext download the file from the local datastore by either from webclient GUI or WINSCP , this file can be later shared with respective OS vendor for debugging.\nOption 2 :\nIf the vSAN cluster does not have a local datastore configured . We will need to follow below steps to download the file .\nTo start first Suspend the VM following the steps mentioned in the KB :- https://kb.vmware.com/kb/2005831\nThis will create a vmss and a .vmem file for Debugging\nLog on to the Esxi host using a Putty, where the VM in question is registered and running\nNavigate to Virtual machine directory location for example “cd /vmfs/volumes/vsan:524fcc9a876dcefd-be39dfd52bd37e0e/”\nRun a command “cat” against the .vmem file :- cat .vmem\nRetrieve the Object ID information as show in the output below :\n[root@host4:/vmfs/volumes/vsan:524fcc9a876dcefd-be39dfd52bd37e0e/94231059-ce43-435b-58f5-0050560150c2] cat vm-win-crash-5675fa8a.vmem\nObject DescriptorFile version = “1”\nobjectID = “vsan://e6a82759-adc5-24b5-1853-0050560150c2” –\u0026gt; Use the Object ID to copy\nobject.class = “vmem”\nThen run the command :- /usr/lib/vmware/osfs/bin/objtool open -u , see example below : In this case the command is :- /usr/lib/vmware/osfs/bin/objtool open -u e6a82759-adc5-24b5-1853-0050560150c2\n====================================================================================================================================\n[root@host4: /usr/lib/vmware/osfs/bin/objtool open -u e6a82759adc5-24b5-1853-0050560150c2\nObject opened at /vmfs/devices/vsan/ e6a82759-adc5-24b5-1853-0050560150c2 –\u0026gt; Shows opened\n[root@host4:/vmfs/volumes/vsan:524fcc9a876dcefd-be39dfd52bd37e0e/94231059-ce43-435b-58f5-0050560150c2]\n====================================================================================================================================\nThen use “winscp”, logon to the Esxi host, browse to the file path which is /vmfs/devices/vsan/e6a82759-adc5-24b5-1853-0050560150c2 to move the file to the local location or remote location of the machine from which you have logged in .\nOnce the file has been copied over to your local system , you will need to rename the file extension to .vmem so that Microsoft or any vendor can use this file for Debugging purpose .\n","link":"https://blogs.hareeshkg.bid/post/download-vmem-file-from-a-vsan-datastore/","section":"post","tags":[],"title":"Download vmem file from a vSAN datastore"},{"body":"","link":"https://blogs.hareeshkg.bid/series/","section":"series","tags":null,"title":"Series"}]