Exchange 2010 : Cannot add a second DAG member

Hi,

today I will talk about this error :

Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error An error occurred while attempting a cluster operation. Error: Cluster API '"AddClusterNode() (MaxPercentage=100) failed with 0x5b4. Error: This operation returned because the timeout period expired"' failed.. ---> System.ComponentModel.Win32Exception: This operation returned because the timeout period expired

So you will find a looooooot of post about it, but most of time no real answers. You will find good advice like : Check replication network, DNS, order of the NIC in the NIC binding, Services activate on the card, Disable tcp Chimmey, TCPIP Offload…and more and more…cool but at the end I’m pretty sure that you will not be able to add a second Server to your DAG.

No more wait, the root cause is coming from the server. From the NIC configuration of the server. From the Teaming !

OMG ! Yes. I will describe my case.

  • 2 Servers on Windows Server 2012 with Hyper-V (A and B)
  • 3 Servers Exchange 2010 with CAS, HUB and Mailbox (X, Y and Z)
  • the server A has a teaming (windows 2012 teaming) and host the server X
  • the server B has NO teaming and host server Y and Z

At this point it is possible
to create the DAG and had Y and Z but NOT X
to create the DAG and had X but NOT Y or Z

the teaming of windows server 2012 is not really compatible with the Failover Cluster service. The teaming is (by default) in “Switch independent” and the algorithm used is “Hash Address”. This algorithm creates a hash based on components of the packet, and then it assigns packets that have that hash value to one of the available network adapters.

So you have to solutions:
– Remove the Teaming on the network card use by the DAG
– try to use the “Hyper-v Port” (it was not working for me)
– try to recreate the Teaming with Intel or Broadcom drivers

If your exchange servers are on a SCVMM with Hyper-V you can group your Exchange DAG Members on one host and setup them to move together in cas of failure of one physical host. If you do a such thing your dag members will be always on the same physical host. It is for me a best practice if you want to continue to use Teaming with DAG.

 

10 thoughts on “Exchange 2010 : Cannot add a second DAG member

  1. Good grief, this is what I’ve been searching for. I thought I was going crazy having rebuilt my Exchange VMs three times with different names, on different hosts, etc. I had no idea the NIC teaming of the underlying host would be the reason for all my trouble.

    I set the load balance method to “Hyper-V Port” with no standby adapters at first, but ultimately had to set one NIC to standby. That way there was only one NIC active, and one standby. This didn’t work either.

    Ultimately I had to use a single NIC for my Virtual Switch and that was the only way to have it work properly.

  2. GOOD GRIEF! I have been working on this for hours and checking all the things that everyone else said (like you mentioned), but sure enough when I moved it to a standalone NIC it worked. The only thing I will mention is that all my teaming is done through Broadcom’s network utility, so I’m not sure that option 3 will work in all cases.

    But still – this has got to be a known issue that Microsoft could put in the server requirements – “By the way, no Hyper-V teamed NICs please. Thank you.”

  3. Hello,
    i also spent many hours on the same problem in Exchange 2013.
    I have a Hyper-V 2012 R2 preview cluster with two nodes.
    I configured the two network adapters in teaming (switch independent, Hyper-V port).
    With this configuration, I found that Exchange 2013 DAG communication worked fine through the Microsoft Failover Cluster Virtual Adapter (ipv6) only when the two Exchange 2013 VMs were hosted on the same Hyper-V host.
    But i didn’t know why.
    After reading this article, i removed the teaming.
    I confirm that DAG communication is working fine now, even when Exchange 2013 VMs are hosted on separate Hyper-V node.
    Thx
    Stef

    • you should pass here immediately an not wait many hours ;)…i’m joking. The issue with Teaming + Hyper-v and the DAG is an algorithm problem. Maybe I will do an article to explain in detail the issue but it could be not very interesting for a lot of people. The fact is : you need teaming with DAG ? I’ m pretty sure you can use the driver from Intel or from your network card to do it and it will work.
      Moreover your intervention is a proof that Windows 2012 R2 has the same network mechanism than windows 2012.

  4. Microsoft has pubished a KB regarding this problem : http://support.microsoft.com/kb/2872325/en-us

    I reconfigured Windows 2012 teaming, disabled “Microsoft Failover Cluster Virtual Adapter Performance Filter” on Hyper-V hosts network adapters.

    Cluster communication is working fine now on my clustered guests.
    I can see established connections on port 3343 in ipv6 through microsoft virtual adapters

    Regards

  5. thank you. Now It Works Well form me, after disabling “Microsoft Failover Cluster Virtual Adapter Performance Filter” on Hyper-V hosts network adapters including the team adapter.

  6. my fix was to uninstall Failover Clustering and the use add-databaseavailibilitygroupserver, which installed the clustering role again and added the server to the DAG successfully

Leave a Reply

Your email address will not be published. Required fields are marked *

*