Hi All –
Recently in our infrastructure we have deployed ESXi on CISCO UCSC-C240-M4SX model hardware and used the same to migrate VM workload from old hardwares which has reached End of Life.
During the migration, we have experienced slowness in data transfer. Speed of Data transfer was approximately 16MBps & time taken for VM migration with 250GB used space was near to 10Hrs. Migration of VM workload was happening with in same datacenter & Management network of ESXi hosts are part of same VLAN.
In our past experience, similar type of 250GB-VM migration within Datacenter expected to complete with in 2Hrs max. So, we have decided to trace the cause of slowness experienced with this CISCO Model servers.
At first, checked the speed & connectivity status of network ports used by management network. No Issues found.
Next moved further to performance statistics of Local SCSI controller since the VM data migration targeted to local raid store. Performance chart indicates latency was consistently above 20ms while performing the migration write process. This made us to suspect there might be some issue with SCSI controller.
CISCO 12G SAS Modular Raid Controller with 1GB Flash Backed Write Cache found to be available with this model servers. 24 x 300GB SAS Magnetic Disks attached to it & they were partitioned as [ 2 Disks – Raid 1 Store & Remaining Disks – Split in to 3 No of Raid 5 Stores ]. Even though the controller has flash backed cache, raid write policy found set as “Write through” by default. This was the cause which created slowness during VM migration.
Lets see what is Flash Backed Write Cache & Raid write policy setting ??
Flash Backed Write Cache is a component which helps controller to move data available in cache memory to non-volatile flash device in case of unexpected power failure.
White paper from HP explaining FBWC for reference – https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c00687518
Raid write policy is the setting which instructs controller to use cache memory for write operation or not.
Basically there are two settings available – Write through & Write Back.
Write through – Data write happen directly to disks. Controller sends data transfer completion signal to host once disks receive all the data in transaction.
Write Back – Data write happens in cache memory of controller at first and then it is moved to disks. Controller sends data transfer completion signal to host once cache receive all the data in transaction.
We have changed the write policy setting of Raid stores to “Write-Back” since we have flash backed write cache with the SCSI controller. After this change, reinitiated migration of VM with 250GB used space to identify the difference. This time migration speed was around 200MBps and duration took was approximately 23 minutes to complete 🙂
In Essence, CISCO UCSC-C240-M4SX model servers are coming with “Write through” Raid Write Policy setting by default. Hence, suggesting to validate and change them according to your requirement & usage before hosting the workloads.
White paper available in below link for more insight:
Thanks for reading the post and do share your views 🙂
Never Stop Learning !