I’ve Made A Huge Mistake

One way to schedule resources in Exchange 2003 is by using Direct Booking.

http://msexchangeteam.com/archive/2006/02/22/420275.aspx

In Exchange 2007/2010 the resources are now actually special mailbox types combined with the Availability Service too book resources.

If you use Outlook 2000/2002/2003/2007 with Exchange 2003 you have no issues doing Direct Booking as stated here.

http://support.microsoft.com/kb/291616

What if you are on Outlook 2010 and Exchange 2003? The Outlook team assumes you are using Exchange 2007/2010 so direct booking is not enabled by default in the Outlook 2010 client. If a user tries to use Direct Booking as in previous versions of Outlook they will get a bounce back message saying the resource was not booked properly. Clearly confusing to the end user.

The fix, enable the Direct Booking feature in Outlook 2010. It will now work like previous versions of Outlook.

http://support.microsoft.com/kb/982774

Powershell is awesome. We all know it. However if you are still on Exchange 2003, you “sometimes” (read all the time) have to do things that are less optimal as opposed to a one line powershell script. I’ve recently run into a scenario where we’ve had an account that has been compromised and sent out a lot of spam to “many” (read thousands) different users and domains. If it is only a few domains you can easily use ESM, search the queue for that user, Delete with No NDR, you are all done. But if you were like myself and had 1 or 2 messages to thousands of mail queues, this wont cut it. Enter the tool aqadmcli.exe. This little tool can be nabbed at ftp://ftp.microsoft.com/pss/Tools/Exchange%20Support%20Tools/Aqadmcli/aqadmcli.exe and is used just for the task at hand, clearing out SMTP queues from the command line.

Alright so we got our command line tool, let’s let her rip and clean house. If you run adadmcli.exe /? you get a whole list of different usage commands which we wont go into depth here, just how to solve our problem at hand. However how you run it on a front end Exchange server and back end server in a cluster our a little bit different and isn’t completely clear in /?. But let’s dive into the process I used for this.

1.) First things first, disable this account in AD. Their account has been compromised, it’s getting turned off. We’ll get you back online later after this mess is cleared up thank you very much.

2.) Disable outbound mail for your server or servers (front and back). Let’s do our little part of trying not to completely turn the Internet into a spam wasteland if we can help it. We do this by simply going into ESM, expand the server, click on “Queues” and hit the big “Disable Outbound Mail” button. If you are in a Front End/Back End architecture, you would do it on both your Front End server and the Back End server this persons account resides on. We have clean up in multiple places that needs to take place most likely. Your back end may be completely clean but it may also be backed up at this point. After this step, all outbound mail for your organization is disabled.

3.)  Let’s clean up the front end servers first so we can get outbound mail flowing for the other servers that don’t need to be cleaned. Copy over aqadmcli.exe to the front end server. Open up a command prompt and enter aqadmcli.exe and hit enter. Now it’s time to do some cleaning.

4.) Since you are on the front end, you really don’t need to set the server since it defaults to the local host . You can define which queue you want to target but in our case, we are going to clear all of them of ANY mail from this user. So yes if there is actual legitimate mail from this user in these queues, they will be gone as well. To me I’m ok with this, your account just sent out 250k worth of spam clogging up my queues, your mail privileged has been revoked while I work on this.  The actual command we run is the following without the quotes, “delmsg flags=SENDER,sender=username@yourdomain” and hit enter. At this point the tool is doing it’s job, it’s looking through all the queues on this server from any messages from this account and is deleting it. You’ll see it scroll through and how many messages it deleted. I tend to run this a few times just in case there are any messages that are still in transit. After this type quit and enable outbound mail. Do this same process on any other front end servers.

5.) At this point outbound mail for your org is up and clean for your front end servers and all back end servers that don’ t have this compromised account. Now time to clean up the back end server. Once again, copy over aqadmcli.exe and run the program as above. Now since we are on a back end server in a cluster we have to actually set the server we are on and the virtual server that is running on it. We do this by running this command without the quotes, “setserver sn=hostname,vs=number” So for example if my backend server is named xbe01 and the virtual server it is hosting is xvs01 the command would look like “setserver sn=xbe01,vs=1″. After this is set, we run the same command as above to clear out all mail queues from mail from this user, “delmsg flags=SENDER,sender=username@yourdomain”. Once again  I run this a few times make sure all messages that were in transit are completely cleared out. Once this comes back clean we can quit this program and re-enable outbound mail for this back end server. All mail is now flowing outbound for your org.

6.) The last step is to “educate” the user on what happened and change their password/enable their account as part of your normal account enabling process, you do have those right.

And that’s it you are done. It’s little more work than a sweet, sweet powershell command but MUCH faster than actually using ESM. Now sit back and monitor your queues for a bit to make sure everything is ok, that spam didn’t stand a chance.

I ran into a case where the built in Exchange 2003 monitoring tools were misfiring on alerts that a box that wasn’t even set to be monitoring. In ESM if you go to Tools, Monitoring & Status, Notifications, you can configure a Server to monitor various Exchange issues and alert on Warning/Error. What do you do if you are getting them from a box that isn’t listed?

Turn up the logging on WMI  on the phantom box and see what is going on.

1.) Launch wmimgmt.msc

2.) Right-click on WMI Control (local) an select Properties, click the “Logging”  tab

3.) Set the “Logging level” to verbose

These logs will go to C:\windows\system32\wbem\logs

Take a look through these in namespace //./root/cimv2/Applications/Exchange . This may lead you to the alerts that are being generated via WMI. Once you see the SMTPEVENT those are the ones to go after and delete.

To Remove

1. Click Start, run, type Wbemtest then type root\cimv2\applications\exchange and click “Connect” button

2. Click on ‘Enum Classes’, click the Recursive radio button, click OK.

3. Scroll down until you see _FilterToConsumerBinding class.  Double-click on it.

4. Click the “Instances” button on the right hand side.

5. Chose/highlighted the subscriptions with the name you saw in the log and click on the delete button.

Good luck hopefully this will help you get to the bottom of it.

If you are one of the people who still are on Exchange 2003 and have E-discovery compliance for your email, this hot fix is for you.

http://support.microsoft.com/Default.aspx/kb/971660

Goodness right. Some mail might not be getting journaled in some specific situations. Yikes! I know that’s not the thing I’d like to say to the judge who probably doesn’t understand technology let alone how journaling works in Exchange 2003. Anyways, apply this to your back end servers as quick as you can. That is the only one that “REQUIRES” it. But best practices states try to keep your CDO.dll file in the same version number, so go ahead and apply this to your front end servers too. If you run a BES server, that should get this updated to so you get the most updated CDO.dll file and is in line with Exchange.

Fresh from the Exchange team, they will now support Exchange 2003 to query against 2008 R2 DCs. This means that if you are on 2003 and plan on going to 2007 or 2010, you can upgrade your DCs to 2008 R2 and then raise your forest/domain levels to the latest and greatest of 2008 R2 after 2003 has been removed from your environment. This actually saved me 1 upgrade I now no longer have to do.

http://msexchangeteam.com/archive/2009/11/30/453327.aspx

A useful little trick when installing/administrating something that has a potential to eat up storage (think file server, logs for a database) is the fear that the drive will fill up in space without you noticing and you’ll have to add new space immediately. What if this process of adding new space requires downtime or has to be approved? What do you do if you missed the drive space filling up over time and now the system is offline? The answer is a dummy file. I create a semi large dummy file that can be deleted in a pinch if the drive filled up to get everything up and running again on my file server. Think of it as some quick breathing room until you can get the situation under control.

fsutil file createnew dummy.file 10737418240

You can run this at the command line to create your blank large empty file. The last parameter is in bytes, that example is for a 10 GB file.

Just try to gather all some useful info and put it all in one place.

That includes the quorum and MSDTC resource.

http://almostdailytech.com/2009/05/27/changing-the-quorum-disk-in-a-cluster/

How to move Exchange Database (this is similar for moving system path files as well)
http://support.microsoft.com/kb/821915

Moving MS Search

http://almostdailytech.com/2009/06/18/moving-the-ms-search-the-easy-way-and-the-hard-way/

General Cluster

http://msmvps.com/blogs/clusterhelp/archive/2005/08/05/moving-a-cluster-to-a-new-san-original-posted-jul-22-2005.aspx

Borrowing this from another blog which everyone should totally read , http://www.markwilson.co.uk/blog/index.php

Now that MS supports some things to be virtualized but not all aspects, it can be quite confusing depending on your configuration what is supported or not. A simple web form will show you the way.

http://www.windowsservercatalog.com/svvp.aspx?svvppage=svvpwizard.htm

So easy to use, easy to understand and it will show specifically what feature is and is not supported. What is supported is a mystery no more!

Original Post

http://blogs.technet.com/mattmcspirit/archive/2009/07/03/the-svvp-wizard.aspx

Recently I’ve gone through the pleasure of having to move the MS Search instance in an Exchange Cluster to a new disk location. There are 3 options. The easy way, the dangerous way, and finally the hard but manual way.

The first way is the easy way. You basically take the mssearch offline, rename the folder, create a new folder as a mount point, then copy the data into it and start the service. Simple if you are able to keep the same drive location. This has a detailed walkthrough of what I sort of  outlined.

http://support.microsoft.com/kb/938445

The second way I think its much more dangerous to me, since it involes deleting data. Basically if you wanted to simulate that you lost the drive and you need to recreate search you would do the following. Run a script that is listed in this KB that will delete the registry settings in the cluster on that node, stop and start the MS Search, delete the System Attendant and recreate it You will lose any custom settings you have in the System Attendant. Then you basically recreate the System Attedant. Detailed steps can be found here. http://support.microsoft.com/kb/830189/en-us

Wow that seems pretty scary, deleting data and hoping you dont have custom stuff if you don’t actually know it. Seems dangerous to me.
The third way is the manual way. Which is also semi dangerous but at least a controlled danger. I take no responsibility for any of these steps. If you really want this done right call MS and open a support case or at least attempt this in a lab first. This is just a good direction with my fuzzy memory from a few months ago. I’m mostly confused by the passive node aspect of it. If I remember I’ll update it.

First you need to have your new drive set up for where the files are going to go. The MS Search needs to be online for this first part otherwise the registry settings on the local machine DO NOT SHOW UP. This is key. You can test this by taking the Search offline and seeing the keys disappear. You may also want to take the search out of the dependencies so it doesn’t take your entire Exchange Virtual Server Offline. You then want to change the following keys after you backup your registry naturally.

1.)Change the ‘SearchDirectory’ string to corresponding new drive location under -

HKLM\Software\Microsoft\Search\1.0\Applications\ExchangeServer_<EVS name> hive

Change the strings ‘FileName’ and ‘LogPath’ & pointe them to the corresponding new drive location.·

HKLM\Software\Microsoft\Search\1.0\Databases\ExchangeServer_<EVS name>

Change the strings ‘ApplicationPath’ and ‘DefaultProjectPath’ & point them to the corresponding new drive location.

HKLM\Software\Microsoft\Search\1.0\GatheringManager\Applications\ExchangeServer_<EVS name>

Change the string ‘ApplicationPath’ and point it to the corresponding new drive location.

HKLM\Software\Microsoft\Search\1.0\Indexer\ExchangeServer_<EVS name>

Finally check through out the hive to make sure nothing is pointing to the old location
HKLM\Software\Microsoft\Search\1.0”

2.) Then take the MSSearch resource offline. (Note if you try to copy the data while it’s online it will sucesffully copy but it wont come on-line).

3.) Then on all physical nodes you need to change this registry setting.

Change the string ‘ApplicationPath’ and pointed it to the corresponding new location.

HKLM\Cluster\Resources\<GUID>of the MSSearch Resource>\Parameters

4.) Then copy the Exchangeserver_<EVS name> directory from old drive as whole to the new drive in the corresponding path. Rename the folder “_old” in the old path for backup purpose.

5.) Bring the resource online.

There aren’t enough Warnings in the world for you before you do this. Please test this in the lab first if possible.

I recently had to change SANs and one of the disks you need to move is the Quorum. It’s relatively straight forward. After you change the Quorum to the new disk (remember the new disk has to be online in the cluster), then you’ll then need to stop the MSDTC resource, copy the MSDTC folder to the new disc, then delete the MSDTC resource. Then just simply re-create it.

Changing Quorum Disks

http://support.microsoft.com/default.aspx?scid=kb;en-us;280353

Recreate the MSDTC Resource

http://support.microsoft.com/kb/301600/