Testing Exchange Server Updates – Part 1

So. Another update. Another bug. Another pulled update. More irate customers. More headaches for the IT guy/gal. Finally a re-posting of the Update (v2 of course).

What do you do?

Test. Test and test some more.

Then get sign off from management and apply your patches. Use a Change Control process and be prepared for recovery/rebuilding as needed.

How do we do this for corporate systems like email? How can you test to make sure you don’t fall into this same trap?

Test Platform
First we need an environment that mimics as closely as possible our production environment. This does not mean that every single details is duplicated, but we should make a best effort to duplicate what we can duplicate. Here is a short list:

*** Remember to isolate this lab so as to not cause any corruption for Exchange or Active Directory ***

  • Active Directory – create base server and perform an authoritative restore from backup
  • Exchange Servers – restore Exchange using AD config and restore all databases for testing
  • Desktops – if you have a standard image or two, just recreate – same version of Outlook will help immensely
  • Application that relay email or are integrated

The above setup will give us a good baseline for testing most Exchange functionality. Some items that might not be duplicated due to budget or other concerns would be:

  • Load balancing

Test methodology
Once the production Exchange environment has been mimic’d in the lab we can move on to testing. We need the following tools to best test Exchange after an update has ben performed on the lab servers:

  • Outlook client(s)
  • OWA client(s)
  • ActiveSync client(s)
  • PowerShell – Active Directory, Exchange and Quest AD PowerShell Snap-Ins

For Outlook, a test script or set of outlined steps should be used. An example of these steps includes the following:

  • Compose and send a new email and add a user on Exchange 2013 – same server
  • Compose and send a new email and add a name of someone on Exchange 2013 – different server
  • Compose and send a new email and add a name of someone external to the environment (is possible)
  • Move an email to archive
  • View an email in archive
  • Create and meeting and accept/decline the meeting with Outlook, OWA and Mobile phone
  • Set Out of Office – internal only as well as internal / external
  • Verify AutoDiscover works for a new Outlook profile
  • Validate Retention Policies work
  • Validate Policy / Mail Tips work as expected

Your complete test list should be much longer.

Automation – Part One
What can be automated? For test environments we generally do not have the same resources to apply for testing and thus any sort of automation will alleviate the time crunch for testing the patch. What can we test?

Email flow
The simplest, and of course, most obvious testing is mail flow. PowerShell provides us with functionality to do so. Things to test:

  • single recipient with and without an attachment
  • Multiple recipients with and without attachments
  • Message that exceeds any transport limits – does this fail
send-mailmessage -from admin@domain.com -to mailuser1@domain.com -subject "Test for patching 1" -smtpserver $line2.name -body "This is a test, please ignore and or delete this message.  Thanks."

What if we wanted to loop the above process? or even try different recipients for each round of messages?

$mailboxes = get-mailbox
$sender = administrator@domain.com
$servers = get-transportserver 
foreach ($line in $mailbox)
     $recipient = $line.primarysmtpaddress
     
     # Send a message to each transport server
     foreach ($line2 in $servers) {
          send-mailmessage -from $sender -to $recipient -subject "Test for patching 1" -smtpserver $line2.name -body "This is a test, please ignore and or delete this message.  Thanks."
     }

     # Change the sender to be the previous recipient to allow for different senders
     $sender = $recipient
}

We can also loop this further by adding lines 6, 7 and 20 to the above script:

$mailboxes = get-mailbox
$sender = administrator@domain.com
$servers = get-transportserver 

foreach ($line in $mailbox) {
     $i = 0
     do {
          $recipient = $line.primarysmtpaddress
     
     # Send a message to each transport server
          foreach ($line2 in $servers) {
               send-mailmessage -from $sender -to $recipient -subject "Test for patching 1" -smtpserver $line2.name -body "This is a test, please ignore and or delete this message.  Thanks."
          }

     # Change the sender to be the previous recipient to allow for different senders
          $sender = $recipient
          $i++
     }
     # Loop for 50 messages
     while ($i -le 50)
}

Don’t forget to test attachments as well, which can be done with this script below. For this test, I picked two files, one that is 10MB and the other is 150MB. If your connectors are configured for the default size limits, the 150MB attachment should fail. If the connectors have higher limits, you would need a larger file than the connector size to create a failure test:

$mailboxes = get-mailbox
$sender = administrator@domain.com
$servers = get-transportserver 
$attach1 = "C:\Files\SmallFile.PDF"
$attach2 = "c:\Files\LargeFile.PDF

foreach ($line in $mailbox) {
     $i = 0
     do {
          $recipient = $line.primarysmtpaddress
     
     # Send a message to each transport server
          foreach ($line2 in $servers) {
               # 10MB Test attachment
               send-mailmessage -from $sender -to $recipient -subject "Test for patching 1" -smtpserver $line2.name -body "This is a test, please ignore and or delete this message.  Thanks." -attachment $attach1
               # 100MB Test attachment
               send-mailmessage -from $sender -to $recipient -subject "Test for patching 1" -smtpserver $line2.name -body "This is a test, please ignore and or delete this message.  Thanks." -attachment $attach2
          }

     # Change the sender to be the previous recipient to allow for different senders
          $sender = $recipient
          $i++
     }
     # Loop for 50 messages
     while ($i -le 50)
}

Once a final script is created, the testing process should be to schedule the task to run every 1, 5, 10, 30 or 60 minutes to simulate some sort of user traffic as well as to fill up the queues a bit for testing and to generate logs for analysis. User this blog link as an example of how to do this.

There are other SMTP testing methodologies and scripts available from other Exchange MVPS:

  • Paul Cunningham has the Mail Flow Heat Map script that will expose any high latency issues between servers.
  • Test-SmtpConnectivity PowerShell command will test all connectors on a local server and running this command will test all server connectors:

    Get-TransportService | Test-SmtpConnectivity

  • Test-MailFlow is another command that can be used to verify mail flow is working properly:

    Get-TransportService | Test-MailFlow

More Testing
In part two of this series I will cover more tests that can be automated with reporting and validation of the patch’s success.

Further Reading
Test-MailFlow
Test-SMTPConnectivity
Send-MailMessage

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s