VMware Paravirtual SCSI adapter: Is it really that much faster?

I asked the same question myself after reading a best practice guide from Datrium that suggested using the VMware PVSCSI controller instead of the default recommendation of the LSI SAS controller that VMware makes when you create a Windows VM.

Out of curiosity I spun up a new server 2016 VM. 4 Cores 8GB of RAM, and a 100GB drive, hosted on my Datrium storage to find out how much of a difference there was.

For this test I ran during a normal production workload, and used Microsoft DiskSpd with a 16k IO size (my current average for my app servers) to test to see what we would get for results. The specific command I used was

diskspd.exe -b16K -d1800 -h -L -o2 -t4 -r -w50 -c10G C:\io.dat

The first run on the VMware LSI SAS controller resulted in this.

Command Line: C:\Users\cjoyce_admin\Downloads\Diskspd-v2.0.17\amd64fre\diskspd.exe -b16K -d1800 -h -L -o2 -t4 -r -w50 -c10G c:\io.dat

Input parameters:

timespan: 1
-------------
duration: 1800s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'c:\io.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 16384
using random I/O (alignment: 16384)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 4
using I/O Completion Ports
IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time: 1800.00s
thread count: 4
proc count: 4

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 8.35%| 1.84%| 6.50%| 91.65%
1| 8.38%| 1.89%| 6.48%| 91.62%
2| 7.78%| 1.79%| 5.99%| 92.22%
3| 7.39%| 1.60%| 5.79%| 92.61%
-------------------------------------------
avg.| 7.97%| 1.78%| 6.19%| 92.03%

Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 15150776320 | 924730 | 8.03 | 513.74 | 3.888 | 3.175 | c:\io.dat (10240MB)
1 | 15089106944 | 920966 | 7.99 | 511.65 | 3.904 | 3.289 | c:\io.dat (10240MB)
2 | 15108947968 | 922177 | 8.00 | 512.32 | 3.899 | 3.140 | c:\io.dat (10240MB)
3 | 15109013504 | 922181 | 8.01 | 512.32 | 3.898 | 3.086 | c:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 60457844736 | 3690054 | 32.03 | 2050.03 | 3.897 | 3.173

Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7574110208 | 462287 | 4.01 | 256.83 | 3.274 | 2.741 | c:\io.dat (10240MB)
1 | 7539032064 | 460146 | 3.99 | 255.64 | 3.297 | 2.966 | c:\io.dat (10240MB)
2 | 7562526720 | 461580 | 4.01 | 256.43 | 3.297 | 2.861 | c:\io.dat (10240MB)
3 | 7543046144 | 460391 | 4.00 | 255.77 | 3.293 | 2.613 | c:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 30218715136 | 1844404 | 16.01 | 1024.67 | 3.290 | 2.798

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7576666112 | 462443 | 4.01 | 256.91 | 4.501 | 3.448 | c:\io.dat (10240MB)
1 | 7550074880 | 460820 | 4.00 | 256.01 | 4.510 | 3.479 | c:\io.dat (10240MB)
2 | 7546421248 | 460597 | 4.00 | 255.89 | 4.501 | 3.289 | c:\io.dat (10240MB)
3 | 7565967360 | 461790 | 4.01 | 256.55 | 4.503 | 3.389 | c:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 30239129600 | 1845650 | 16.02 | 1025.36 | 4.504 | 3.402
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.000 | 0.000 | 0.000
25th | 1.360 | 2.258 | 1.709
50th | 2.818 | 3.885 | 3.269
75th | 4.481 | 6.093 | 5.443
90th | 6.259 | 8.370 | 7.195
95th | 7.163 | 9.928 | 8.987
99th | 10.090 | 13.425 | 12.593
3-nines | 23.523 | 30.284 | 27.785
4-nines | 47.191 | 52.535 | 49.878
5-nines | 190.339 | 161.402 | 190.339
6-nines | 534.581 | 534.289 | 534.289
7-nines | 545.593 | 535.040 | 545.593
8-nines | 545.593 | 535.040 | 545.593
9-nines | 545.593 | 535.040 | 545.593
max | 545.593 | 535.040 | 545.593

Overall not terrible. Now lets look at what we get when we replace the LSI SAS with a PVSCSI.

Input parameters:

timespan: 1
-------------
duration: 1800s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'c:\io.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 16384
using random I/O (alignment: 16384)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 4
using I/O Completion Ports
IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time: 1800.00s
thread count: 4
proc count: 4

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 7.37%| 1.53%| 5.84%| 92.63%
1| 7.02%| 1.40%| 5.62%| 92.98%
2| 6.35%| 1.25%| 5.10%| 93.65%
3| 6.04%| 1.22%| 4.82%| 93.96%
-------------------------------------------
avg.| 6.70%| 1.35%| 5.35%| 93.30%

Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 15667019776 | 956239 | 8.30 | 531.24 | 3.760 | 2.938 | c:\io.dat (10240MB)
1 | 15743369216 | 960899 | 8.34 | 533.83 | 3.741 | 3.011 | c:\io.dat (10240MB)
2 | 15789637632 | 963723 | 8.37 | 535.40 | 3.730 | 2.841 | c:\io.dat (10240MB)
3 | 15788425216 | 963649 | 8.36 | 535.36 | 3.731 | 2.914 | c:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 62988451840 | 3844510 | 33.37 | 2135.84 | 3.740 | 2.926

Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7831814144 | 478016 | 4.15 | 265.56 | 2.660 | 2.405 | c:\io.dat (10240MB)
1 | 7862943744 | 479916 | 4.17 | 266.62 | 2.640 | 2.538 | c:\io.dat (10240MB)
2 | 7904346112 | 482443 | 4.19 | 268.02 | 2.632 | 2.247 | c:\io.dat (10240MB)
3 | 7881277440 | 481035 | 4.18 | 267.24 | 2.631 | 2.557 | c:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 31480381440 | 1921410 | 16.68 | 1067.45 | 2.641 | 2.440

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7835205632 | 478223 | 4.15 | 265.68 | 4.859 | 3.010 | c:\io.dat (10240MB)
1 | 7880425472 | 480983 | 4.18 | 267.21 | 4.840 | 3.045 | c:\io.dat (10240MB)
2 | 7885291520 | 481280 | 4.18 | 267.38 | 4.831 | 2.946 | c:\io.dat (10240MB)
3 | 7907147776 | 482614 | 4.19 | 268.12 | 4.827 | 2.833 | c:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 31508070400 | 1923100 | 16.69 | 1068.39 | 4.839 | 2.959
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.000 | 0.000 | 0.000
25th | 1.189 | 2.947 | 1.810
50th | 1.868 | 4.126 | 3.120
75th | 3.536 | 6.037 | 4.971
90th | 5.392 | 8.026 | 6.924
95th | 6.269 | 9.628 | 8.417
99th | 9.446 | 13.234 | 12.021
3-nines | 22.655 | 32.422 | 28.825
4-nines | 45.679 | 50.249 | 48.554
5-nines | 158.326 | 159.371 | 159.371
6-nines | 475.470 | 427.329 | 427.329
7-nines | 475.711 | 427.338 | 475.711
8-nines | 475.711 | 427.338 | 475.711
9-nines | 475.711 | 427.338 | 475.711
max | 475.711 | 427.338 | 475.711

So overall we see roughly a 4% performance increase across the board. Not groundbreaking numbers, however if you’re trying to squeeze every last drop of performance out of your VMs this could be a big step in the right direction.

Speaking of squeezing every last drop, lets see what happens when we test against a ReFS formatted disk.

Command Line: C:\Users\cjoyce_admin\Downloads\Diskspd-v2.0.17\amd64fre\diskspd.exe -b16K -d1800 -h -L -o2 -t4
10G E:\io.dat

Input parameters:

timespan: 1
-------------
duration: 1800s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'E:\io.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing mix test (read/write ratio: 50/50)
block size: 16384
using random I/O (alignment: 16384)
number of outstanding I/O operations: 2
thread stride size: 0
threads per file: 4
using I/O Completion Ports
IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time: 1800.02s
thread count: 4
proc count: 4

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 8.65%| 1.62%| 7.03%| 91.35%
1| 8.69%| 1.49%| 7.20%| 91.31%
2| 7.83%| 1.35%| 6.47%| 92.17%
3| 7.43%| 1.36%| 6.07%| 92.57%
-------------------------------------------
avg.| 8.15%| 1.46%| 6.69%| 91.85%

Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 18047041536 | 1101504 | 9.56 | 611.94 | 3.263 | 2.708 | E:\io.dat (10240MB)
1 | 18078842880 | 1103445 | 9.58 | 613.02 | 3.258 | 3.004 | E:\io.dat (10240MB)
2 | 18066751488 | 1102707 | 9.57 | 612.61 | 3.260 | 2.712 | E:\io.dat (10240MB)
3 | 18132910080 | 1106745 | 9.61 | 614.85 | 3.248 | 2.727 | E:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 72325545984 | 4414401 | 38.32 | 2452.42 | 3.257 | 2.791

Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 9020080128 | 550542 | 4.78 | 305.85 | 2.762 | 2.399 | E:\io.dat (10240MB)
1 | 9030025216 | 551149 | 4.78 | 306.19 | 2.760 | 2.927 | E:\io.dat (10240MB)
2 | 9041592320 | 551855 | 4.79 | 306.58 | 2.759 | 2.342 | E:\io.dat (10240MB)
3 | 9050865664 | 552421 | 4.80 | 306.90 | 2.752 | 2.479 | E:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 36142563328 | 2205967 | 19.15 | 1225.53 | 2.758 | 2.547

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 9026961408 | 550962 | 4.78 | 306.09 | 3.764 | 2.899 | E:\io.dat (10240MB)
1 | 9048817664 | 552296 | 4.79 | 306.83 | 3.754 | 2.998 | E:\io.dat (10240MB)
2 | 9025159168 | 550852 | 4.78 | 306.03 | 3.762 | 2.954 | E:\io.dat (10240MB)
3 | 9082044416 | 554324 | 4.81 | 307.96 | 3.742 | 2.870 | E:\io.dat (10240MB)
-----------------------------------------------------------------------------------------------------
total: 36182982656 | 2208434 | 19.17 | 1226.90 | 3.756 | 2.931
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.267 | 0.297 | 0.267
25th | 1.252 | 1.773 | 1.403
50th | 2.019 | 3.097 | 2.618
75th | 3.724 | 5.038 | 4.275
90th | 5.581 | 6.998 | 6.240
95th | 6.395 | 8.584 | 7.525
99th | 9.641 | 12.213 | 11.021
3-nines | 20.505 | 26.232 | 23.305
4-nines | 42.971 | 45.559 | 44.280
5-nines | 238.498 | 175.573 | 204.921
6-nines | 502.382 | 359.149 | 435.862
7-nines | 547.128 | 547.124 | 547.128
8-nines | 547.128 | 547.124 | 547.128
9-nines | 547.128 | 547.124 | 547.128
max | 547.128 | 547.124 | 547.128

With a ReFS formatted disk on top of PVSCSI we see a 17% increase!

So if your applications support it, and you truly want to squeeze every last drop out of your storage, ReFS and PVSCSI is the combination to go with!

Advertisements

Windows Server Backup using Powershell.

I needed a script to be able to do an on demand backup of a windows server without installing 3rd party software on it. The idea is that physical or virtual, if I needed a quick backup of a box, including a log for auditing, I could have a click to run solution.

Here is that solution. It works for sure on Server 2012, 2012 R2, and 2016 Boxes. 2008 / R2 may need some additional tweaking.

#################################################################
#   Windows Server Backup                                       #
#   Created by - Cameron Joyce                                  #
#   Last Modified - May 2nd 2017                                #
#################################################################
# This script is used to do an on demand backup of a windows server (server 2008 or newer). 

# Variables
$date = (Get-Date).ToString('MM-dd-yyyy')
$time = (Get-Date).ToString('MM-dd-yyyy HH:mm:ss')
$hostname = $env:COMPUTERNAME
$backupserver = "your.server.fqdn"
$osversion = (Get-CimInstance Win32_OperatingSystem).version
$neterr = $false

# Setup folder and logfile.
If(Test-Connection -ComputerName $backupserver -count 1 -Quiet){
    # Try / Catch block for WMI errors. A client that passes Test-Connection may not have PSRemoting enabled and will error. This will handle that.
    Try{
        $ErrorActionPreference = "Stop"
        If(!(Test-Path "\\$backupserver\wsbackups\$hostname")){
            New-Item "\\$backupserver\wsbackups\$hostname" -Type Directory
        }
    }
    Catch [System.Management.Automation.Remoting.PSRemotingTransportException]{
        Write-Warning "Failed connection to backup server."
        $neterr = $true
        If($neterr -eq $true){
            Send-MailMessage -From "smtp@address.com" -To "rcpt@address.com" -Subject "$hostname failed scripted backup. Unable to connect to network storage." -Body "$hostname failed backup because it was unable to connect to '
            $backupserver. Please check the connections and try again." -SmtpServer "srvr.server.com"
        }
        Break
    }
    Finally{
        $ErrorActionPreference = "Continue"
    }
}

# Create Directories and logs.
If(!(Test-Path "\\$backupserver\wsbackups\$hostname\$Date")){
    New-Item "\\$backupserver\wsbackups\$hostname\$Date" -Type Directory
}
$logfile = "\\$backupserver\wsbackups\$hostname\$date\$hostname.$date.txt"

# Verify WSB is installed and load modules. If it is not installed, install it.
Import-Module ServerManager
$bkup = Get-WindowsFeature *backup*
# This if loop contains the commands for install on both 2008 - 2012 as well as server 2016.
If($bkup.InstallState -like "Available"){
    Write-Host "Installing windows server backup role."
    Write-Output "Installing windows server backup role." | Out-File $logfile -append
    If($osversion -like "6.3*" -or "10*"){
    Add-WindowsFeature -Name Windows-Server-Backup -Restart:$false
    }
    Else{Add-WindowsFeature -Name Backup-Features -IncludeAllSubFeature:$true -Restart:$false}
}
Else{
    Write-Host "Server backup is already installed."
    Write-Output "Server backup is already installed." | Out-File $logfile -append
}

# Execute Backup.
Write-Output "Starting Backup at $time" | Out-File $logfile -append
& cmd.exe /c "wbadmin start backup -backupTarget:\\$backupserver\wsbackups\$hostname\$date -allCritical -systemState -vssFull -quiet" | Out-File $logfile -Append
Write-Output "Backup completed at $time" | Out-File $logfile -append

# Look for backup errors.
$eventid4 = $false
$eventlist = Get-WinEvent -logname Microsoft-Windows-Backup | Where {$_.timecreated -gt (Get-Date).Addminutes(-5)} | Select Message
Foreach($line in $table){
    If($line -like "The backup operation has finished successfully."){
        $eventid4 = $true
    }
}

# Send success / failure email.
If($eventid4 -eq $true){
    Write-Host "Backup Success!"
    Send-MailMessage -From "$hostname@domain.com" -To "rcpt@domain.com" -Subject "$hostname has successfully backed up." -Body "Review attachment for backup log." -Attachments "$logfile" -SmtpServer "smtp.server.com"
}
ElseIf($eventid4 -eq $false){
    Write-Host "Backup Failed"
    Send-MailMessage -From "$hostname@domain.com" -To "rcpt@domain.com" -Subject "$hostname has failed backedup." -Body "Review attachment for backup log." -Attachments "$logfile" -SmtpServer "smtp.server.com"
}

PowerShell script to fix VSS errors.

We’ve all had vss writer issues during backups. And many of us have all used the MS technet article to re-register those VSS writers.

Well I had to do that today, and figured I would build a PS script to take care of that so I don’t have to go googling for that article in the future.

#################################################
#   Volume Snapshot Service Repair              #
#   Created by - Cameron Joyce                  #
#   Last Modified - Apr 27 2017                 #
#################################################
# This script is used to repair Microsoft VSS on servers that are failing backups.

# Set Location
sl "C:\windows\system32"

# Stop Services
If((Get-Service -name vss).Status -eq "Running"){
    Stop-Service -Name vss -force
    If(!((Get-Service -name vss).Status -eq "Stopped")){
        Write-Host = "VSS Service failed to stop. Stop manually and re-run script"
        Break
    }
}
If((Get-Service -name swprv).Status -eq "Running"){
    Stop-Service -Name swprv -force
    If(!((Get-Service -name vss).Status -eq "Stopped")){
        Write-Host = "Shadow Copy Provider Service failed to stop. Stop manually and re-run script"
        Break
    }
}

# Re-Register DLLs for VSS
regsvr32 /s ole32.dll
regsvr32 /s oleaut32.dll
regsvr32 /s vss_ps.dll
regsvr32 /s /i swprv.dll
regsvr32 /s /i eventcls.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s vssui.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml4.dll
vssvc /register 

# Start Services
Start-Service vss
Start-Service swprv
If(!((Get-Service -name vss).Status -eq "Running")){
    Write-Host = "VSS Service failed to start. Start service manually."
}
If(!((Get-Service -name swprv).Status -eq "Running")){
    Write-Host = "Shadow Copy Provider Service failed to start. Start service manually."
}

Automatically clean up VMware snapshots using PowerCLI.

Something that every VMware admin who also uses Veeam has had to deal with once if not multiple times is Veeam not cleaning up snapshots properly which then leads to broken snapshot chains, leading to VMDK corruption, and finally leading to an admin crying into his / her bourbon realizing that “no the VM isn’t coming back up, no I can’t consolidate the snap chain and recover it, and no there haven’t been backups in $n days to recover from.” A process I’ve put into my own environment to prevent this is a simple PowerCLI script that looks for all snapshots over 24 hours old, and removes them. VMware recommends that you never have more than 3 snapshots in a chain and those should never be over 72 hours old from a performance standpoint. Personally I agree completely with that. Snapshots should only be used before making a big change so you can quickly roll back, not something that you create and then live off of.

This script requires that the VMware PowerCLI modules are installed on the system you run it from.

#################################################################
#   Remove all snapshots from vSphere from the last 24 Hours    #
#   Created by - Cameron Joyce                                  #
#   Last Modified - Jun 19 2017                                 #
#################################################################
# This script uses PowerCLI to remove all snapshots from virtual machines that are 24 hours old. 

# Load all VMware Modules, and set PowerCLI config.
Get-Module -ListAvailable VM* | Import-Module

# Connect to vSphere vCenter Server.
Try{
    connect-viserver -server your.vmware.server -user administrator@vsphere.local -Password Password
}
Catch{
    Write-Host "Failed Connecting to VSphere Server."
    Send-MailMessage -From "" -To "server@domain.com" -Subject "Unable to Connect to VSphere to clean snapshots" -Body `
    "The powershell script is unable to connect to host your.vmware.server. Please investigate." -SmtpServer "smtp.server.com"
    Break
}

# Variables
$date = get-date -f MMddyyyy
$logpath = "C:\Scripts\Script_Logs"

# Verify the log folder exists.
If(!(Test-Path $logpath)){
    Write-Host "Log path not found, creating folder."
    New-Item $logpath -Type Directory
}

# Get all snapshots older than 24 hours, remove them.
If((get-snapshot -vm *) -ne $null){
    $snapshotlist = get-snapshot -vm * | select VM, Name, SizeMB, @{Name="Age";Expression={((Get-Date)-$_.Created).Days}}
    Write-Host "Current Snapshots in Dallas vSphere"
    Write-Output $snapshotlist
    Write-Output "Snapshots existing before cleanup" | Out-File $logpath\Snapshots_$date.txt -Append
    Write-Output $snapshotlist | Out-File $logpath\Snapshots_$date.txt -Append
}

# Check to make sure that all snapshots have been cleaned up.
If((get-snapshot -vm *) -ne $null){
    get-snapshot -vm * | Where-Object {$_.Created -lt (Get-Date).AddDays(-1)} | Remove-Snapshot -Confirm:$false
    $snapshotlist = get-snapshot -vm * | select VM, Name, SizeMB, @{Name="Age";Expression={((Get-Date)-$_.Created).Days}}
    Write-Host "Current Snapshots in Dallas vSphere after cleanup"
    Write-Output $snapshotlist
    Write-Output "Snapshots existing after cleanup" | Out-File $logpath\Snapshots_$date.txt -Append
    Write-Output $snapshotlist | Out-File $logpath\Snapshots_$date.txt -Append
}
Else{
    Write-Output "No Snapshots to clean up." | Out-File $logpath\Snapshots_$date.txt -Append
}

# Send snapshot log to email.
$emailbody = (Get-Content $logpath\Snapshots_$date.txt | Out-String)
Send-MailMessage -From "server@domain.com" -To "user@domain.com.com" -Subject "Daily vSphere snapshot cleanup report" -Body $emailbody -SmtpServer "smtp.server.com"

# Exit VIM server session.
Try{
    disconnect-viserver -server your.vmware.server -Confirm:$false
}
Catch{
    Write-Host "Failed disconnecting from VSphere."
    Send-MailMessage -From "server@domain.com" -To "user@domain.com" -Subject "Disconnection from VSphere Failed" -Body `
    "The powershell script is unable to disconnect from VSphere. Please manually disconnect" -SmtpServer "smtp.server.com"
}

# Cleanup Snapshot logs older than 30 days.
gci -path $logpath -Recurse -Force | Where-Object {!$_.PSIsContainer -and $_.LastWriteTime -lt (Get-Date).AddDays(-30)} | Remove-Item -Force

There is more logic in here for sending email alerts than actual VMware commands, however when this is running automated from a PS job, it is super helpful to have all the emails.

Create AES secure passwords for use in PowerShell scripting.

Something that I’ve always wanted to get away from in my scripting is leaving passwords in plain text. It fails audits and is just generally insecure and needs to be avoided at all costs. A solution I’ve come up to deal with this so far is to generate a secure key and password hash using AES. Now of course the problem with this is that you still need to secure your keys as they can be used to decrypt your hash into plain text, however the same can be said for PGP or any other reversible encryption.

#################################################################
#   Generate AES Secured Password for Scriptng                  #
#   Created by - Cameron Joyce                                  #
#   Last Modified - Dec 25 2016                                 #
#################################################################
# This script is used to generate a secured AES key and password string for use with other automation. 

# Variables
$Key = New-Object Byte[] 32   # AES Key. Sizes for byte count are 16 (128) 24 (192) 32 (256).
$UnSecPass = Read-Host "Enter Password you wish to use"
$PassName = Read-Host "Enter a filename for the password file."
$SecPass = "$UnSecPass" | ConvertTo-SecureString -AsPlainText -Force
$PasswordFile = "$env:Userprofile\Downloads\$PassName.txt" # OutFile Path for encrypted password.
$KeyFile = "$env:Userprofile\Downloads\$PassName.AES.key" # Path to Generated AES Key.

# Create Random AES Key in length specified in $Key variable.
[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($Key)

# Export Generated Key to File
$Key | out-file $KeyFile

# Combine Plaintext password with AES key to generate secure Password.
$SecPass | ConvertFrom-SecureString -key $Key | Out-File $PasswordFile

How do we use this in real life though? Well here is an excerpt from another script I wrote to install the SCCM agent using my credentials without being logged in.

# Create secure credentials.
$User = "domain\username"
$PasswordFile = "C:\password"
$KeyFile = "C:\key"
$key = Get-Content $KeyFile
$MyCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $User, (Get-Content $PasswordFile | ConvertTo-SecureString -Key $key)

# Using the $MyCredential in practice.
Start-Process Powershell -ArgumentList "-File C:\Temp\SCCM_Agent_Install.ps1 -Verb runAs -Credential $MyCredential"

There is much to be improved on here, however it satisfied my audit requirements and makes me considerably more comfortable that I don’t have a folder full of .ps1 files on a server with my password in plain text.

Cleaning up Lync ADSI attributes for all users in Active Directory for Office 365 migration using PowerShell.

Working through the same migration as my last blog post I ran into a second issue. The client is currently running an on premise Lync 2010 server and they want to migrate all of their users to Skype for Business in Office 365. The users are just using Lync for basic IM, and aren’t using any of the conferencing, or SIP features. Traditionally the way you would move users is to follow the guidelines for a Lync 2010 Hybrid deployment (Microsoft KB explaining how to do this) however that is a lot of effort if all you want is your users in SFB and you don’t care about their data.

If you just license the users in the Office 365 portal you will notice that the license applies, however it doesn’t actually create the Skype user, and the Skype user isn’t able to sign into the Skype client. This is because they still have their on premise Lync Server attributes in their AD account. You will need to clear these out and re-run a Azure AD Sync Delta sync to allow the user accounts to create. Given that going through each user’s ADSI Attributes and clearing them by hand is super tedious, here is a simple PS Script to fix this for you.

#####################################################
#   Remove msRTCSIP Attributes from all users.      #
#   Created by - Cameron Joyce                      #
#   Last Modified - Mar 05 2017                     #
#####################################################
# This script will remove the msRTCSIP attributes from all users in ActiveDirectory. This is meant to be used in an
# Office 365 migration in which you have an on premise lync server, however do not plan to do a hybrid migration to
# migrate users. 

# Variables
$users = Get-ADUser -Filter *

# Foreach user in AD, set all attributes to $null
foreach($user in $users){
    $ldapDN = "LDAP://" + $user.distinguishedName
    $adUser = New-Object DirectoryServices.DirectoryEntry $ldapDN
    $adUser.PutEx(1, "msRTCSIP–DeploymentLocator", $null)
    $adUser.PutEx(1, "msRTCSIP-FederationEnabled", $null)
    $adUser.PutEx(1, "msRTCSIP-InternetAccessEnabled", $null)
    $adUser.PutEx(1, "msRTCSIP-Line", $null)
    $adUser.PutEx(1, "msRTCSIP-OptionFlag", $null)
    $adUser.PutEx(1, "msRTCSIP-PrimaryHomeServer", $null)
    $adUser.PutEx(1, "msRTCSIP–PrimaryUserAddress", $null)
    $adUser.PutEx(1, "msRTCSIP–UserEnabled", $null)
    $adUser.PutEx(1, "msRTCSIP-UserPolicies", $null)
    $adUser.SetInfo()
}

Diagnosing and fixing slow migration times to Office 365 from Exchange 2010.

I’m currently working on an Office 365 migration for a client and have been seeing extremely long migration times for mailboxes. We are talking 30+ hours to move a single 20GB mailbox. The client has a 100Mbps symmetrical fiber pipe, so we know that isn’t the issue, and while their hypervisor and SAN setup aren’t the highest performing we should be seeing better performance than that.

First thing we have to do is actually figure out how bad the situation is. To do so we use this “You had me at EHLO” blog post to figure out what is going on. You need to download this TechNet gallery script and save it locally. Next open up an administrative PowerShell window and change directory to the Directory the script you just downloaded and run the following commands (or if you want, dump them into PowerShell ISE and run them from there.

# Enable unsigned scripts.
Set-ExecutionPolicy Unrestricted

# Create the ProcessStats function.
# Notice the space between the two periods.
# This is important. It is Period space Period Slash.
. .\AnalyzeMoveRequestStats.ps1

# Connect to MSOL Service.
Connect-MsolService

# Set the variables
$moves = Get-MoveRequest | ?{$_.Status -ne 'queued'}
$stats = $moves | Get-MoveRequestStatistics –IncludeReport

# Generate your report.
ProcessStats -stats $stats -name ProcessedStats1

After running this, my results looked like this.

Name                            : ProcessedStats1
StartTime                       :
EndTime                         : 3/6/2017 5:29:27 PM
MigrationDuration               : 1 day(s) 04:08:33
MailboxCount                    : 61
TotalGBTransferred              : 302.20
PercentComplete                 : 92.01
MaxPerMoveTransferRateGBPerHour : 1.75
MinPerMoveTransferRateGBPerHour : 0.19
AvgPerMoveTransferRateGBPerHour : 1.05
MoveEfficiencyPercent           : 80.05
AverageSourceLatency            : 1,469.07
AverageDestinationLatency       : 929.00
IdleDuration                    : 214.40 %
SourceSideDuration              : 71.68 %
DestinationSideDuration         : 14.10 %
WordBreakingDuration            : 7.14 %
TransientFailureDurations       : 0.91 %
OverallStallDurations           : 0.82 %
ContentIndexingStalls           : 0.00 %
HighAvailabilityStalls          : 0.00 %
TargetCPUStalls                 : 0.77 %
SourceCPUStalls                 : 0.05 %
MailboxLockedStall              : 0.00 %
ProxyUnknownStall               : 0.00 %

So our problem is the on premise server causing the problem. We can see in both the AverageSourceLatency and the SourceSideDuration for the idle timer that they are significantly higher than the target. We also notice that we are only transferring 1.75GB per hour which is again horrible.

The resolution for all of this is to increase the MaxActiveMovesPerTargetMDB and MaxActiveMovesPerTargetServer settings from their default of 2, and 5 respectively to anything between 10 and 100. Personally I set mine to 20. Tony Redmond does an excellent job of explaining what these settings do, and how to modify them in his blog post here. Additionally he explains why Microsoft sets them so low to begin with. The TL;DR is that by setting these settings higher you use significantly more CPU and Disk IO on your CAS servers, which in smaller environments can cause disruptions in services to your users, and can also overwhelm your CAS to the point where it can’t keep up with the number of migrations it is running and they time out. All the articles and TechNet forum posts I’ve seen have suggested using a setting of 10 for these, which I agree is a good starting point, then you can tune your server from there. Another note on this is that you need to modify each CAS server in your environment and restart the Exchange Mailbox Replication service after modifying the config, on each server so that this will take effect.