Hybrid Configuration Wizard hanging on Adding Federated Domain or failing to verify DNS TXT records

Today I was delayed a little bit when adding hybrid domains in the Hybrid Configuration Wizard. We added the TXT records as requested and the hybrid configuration wizard kept on showing a question mark when trying to verify the domain ownership.

One of the errors in the Hybrid Configuration log (which can be found under C:\Users\<username>\AppData\Roaming\Microsoft\Exchange Hybrid Configuration) was the following:

HCW 0000 – PowerShell failed to invoke ‘Set-FederatedOrganizationIdentifier’: An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information “An error occurred accessing Windows Live. Detailed information: “Unable to connect to the remote server”.”. {CategoryInfo={Activity=Set-FederatedOrganizationIdentifier,Category=InvalidResult,Reason=ProvisioningFederatedExchangeException,TargetName=,TargetType=},ErrorDetails=,Exception=An error occurred while attempting to provision Exchange to the Partner STS. Detailed Information “An error occurred accessing Windows Live. Detailed information: “Unable to connect to the remote server”.”.,FullyQualifiedErrorId=[Server=EXCH01,RequestId=6daca869-e047-4f04-bc9d-bab9b908c323,TimeStamp=9/22/2017 11:53:46 AM] [FailureCategory=Cmdlet-ProvisioningFederatedExchangeException] 632075DC,Microsoft.Exchange.Management.SystemConfigurationTasks.SetFederatedOrganizationIdentifier}

Again a connection issue… my first hunch: it’s the network!!

…..No… not really, I wouldn’t dare.

2017-09-22_15h10_22

But I did suspect security though and in many cases it’s the same person or team managing it…

We had already added the Office 365 IP Ranges (but only those specifically for Office 365, not the ones for Sway or Skype for Business etc) to the exclusions for the Firewall prior to starting the wizard but I noticed in the logging of the HCW that it was trying to reach an IP over port 443 that was not included in the Office 365 IP Ranges. When checking the source of that IP it seems it was from an Azure DNS server.

Not the most elegant solution but providing full internet access to the server just for that time and purpose solved the problem for us. I was immediately able to add the Federated Domains through the HCW.

Great!

Advertisements

Exchange OWA / ECP – HTTP 500

The website is under maintenance

Hi there,

Not infrequently I bump into a situation whereby the Exchange Admin Center doesn’t want to open and a HTTP 500 Internal Server Error is displayed:

2017-07-14_11h45_09

There are many possible causes and I’ll list some of the things you can check and which have solved the problem for me in the past:

  • One technet article tells you to run ASP.NET RegIIS or to mess with ADSIEdit but please note the following before you try this:
    • ASP.NET RegIIS will not work depending on the version of your operating system.
    • I don’t advise messing with ADSIEdit in general. If you do, make a backup. Secondly, the ADSIEdit solution is only applicable if your HTTP 500 error is accompanied by event ID 1309 in your Application log.

So on to the more likely solutions (if you don’t have event ID 1309 in your Application log):

  • Does the user opening the EAC have a mailbox on the server? If not, try creating one via the shell.

 

  • Is the database containing the mailbox mounted?
    • If not, try to mount it via the Exchange Management Console:
Get-MailboxDatabase "DatabaseName" | Mount-Database

 

  • Is there a problem with one of the system mailboxes?
    • Check if they are present and enabled:
Get-Mailbox -Arbitration | fl Name,IsMailboxEnabled
  • If they are not present, you might have to run your Exchange setup again using the /PrepareAD switch

 

  • If they are present but disabled, enable them using the following command:

For one mailbox:

Enable-Mailbox "SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}" -Arbitration

For all your system mailboxes:

Get-Mailbox -Arbitration | Enable-Mailbox

 

  • Lastly, and if none of the checks/solutions above help, you can try to run the UpdateCas.ps1 PowerShell script. It will restore the bits and pieces of your OWA and ECP virtual directory. The script can be found in the C:\Program Files\Microsoft\Exchange Server\V15\Bin folder. Don’t forget to do an IISReset afterwards (warning: IISReset will cause interrupted connections for your users!)

 

Hope it helps!

SCOM EventID 1008 – Some Client Access test cmdlets failed to run – Exchange 2010

An Outlook Web App connectivity (External) transaction failure occurred. The test credentials can’t be used to test Outlook Web App.

Another failure with lots of interesting (really?) information reported by SCOM today:

< DataItem type =" System.Event.LinkedData " time =" 2017-07-12T08:19:02.8281444Z " sourceHealthServiceId =" 95593a8a-4b31-1a88-4db9-7237206729ba " >

  < EventOriginId > {177a6abb-36be-4e76-8124-03fbd96891a7} 
  < PublisherId > {ce3d4782-7e39-74af-02dd-42cfdae97dec} 
  < PublisherName > ClientAccessTestConfigService.DependsOn.ClientAccessTestConfig.AvailabilityReverseRollup </ PublisherName >
  < EventSourceName > ClientAccessTestConfigService.DependsOn.ClientAccessTestConfig.AvailabilityReverseRollup </ EventSourceName >
  < Channel />
  < LoggingComputer />
  < EventNumber > 500 </ EventNumber >
  < EventCategory > 0 </ EventCategory >
  < EventLevel > 1 </ EventLevel >
  < UserName > NT AUTHORITY\SYSTEM </ UserName >
  < Params />

< EventData >
< CorrelatedContext >
< RootCause >

  < EntityName > Client Access Test Configuration - SecondSiteName 
  < HealthCategory > Availability </ HealthCategory >
  < HealthState > Yellow </ HealthState >
  < MonitorType > Khi 
  < MonitorName > KHI: Some Client Access test cmdlets failed to run - SecondSiteName <!-- MonitorName >
  < MonitorId > {376ef747-f30e-ceef-f8de-4778462bec38} 
  < ProblemTime > 9/07/2017 12:01:47 </ ProblemTime >

< Description >

  < d > There might be a configuration problem in AD site 'SecondSiteName'. Please check the Context field of the alert to see the details. </ d >

  </ Description >
  < AlertName > Some Client Access test cmdlets failed to run 

  < EscalationTeam > Web Services </ EscalationTeam >
  < EntityType > Microsoft.Exchange.2010.Organization </ EntityType >
  < EntityName > DOMAIN - DOMAIN.LOCAL </ EntityName >
  < EntityName > Client Access Test Config - EXCH2010 (Client Access) - SecondSiteName 
  < Computer > EXCH2010.DOMAIN.LOCAL </ Computer >
  < HealthCategory > Availability </ HealthCategory >
  < HealthState > Yellow </ HealthState >
  < MonitorType > Khi 
  < MonitorName > KHI: An Outlook Web App connectivity (External) transaction failure occurred. The test credentials can't be used to test Outlook Web App. 
  < MonitorId > {314262da-3e03-d792-5909-00120d773307} 
  < ProblemTime > 9/07/2017 12:09:16 </ ProblemTime >
< Description >

  < d > The test mailbox was not initialized. Run new-TestCasConnectivityUser.ps1 to ensure that the test mailbox is created. </ d >

  < d > Detailed information: </ d >
   < d > Local Site:SecondSiteName </ d >
    < d > [Microsoft.Exchange.Monitoring.CasHealthUserNotFoundException]: The user wasn't found in Active Directory. UserPrincipalName: extest_6784ccd54b6f4@DOMAIN.LOCAL. Additional error information: [System.Security.SecurityException]: Logon failure: unknown user name or bad password.
  < d > 

Diagnostic command: "Test-OwaConnectivity -TestType:External -MonitoringContext:$true -TrustAnySSLCertificate:$true -LightMode:$true" 
 
< d > EventSourceName: MSExchange Monitoring OWAConnectivity External 
  </ Description 
  < EventSource > MSExchange Monitoring OWAConnectivity External 
  < EventID > 1008 </ EventID >

 

The main cause seems to stem from one particular line:

The user wasn't found in Active Directory. UserPrincipalName: extest_6784ccd54b6f4@DOMAIN.LOCAL

I verified this and indeed this user was not present. Why? Don’t ask me!

Once more, SCOM was so kind as to provide the solution for us:

"The test mailbox was not initialized. Run new-TestCasConnectivityUser.ps1 to ensure that the test mailbox is created"

Which I tried but… Exchange didn’t let me. Another error appeared:

CreateTestUser : Mailbox could not be created. Verify that OU ( Users ) exists and that password meets complexity requirements.

2017-07-13_13h09_26

Whatever OU (using the -OU parameter) or Password I chose, it just kept on throwing the same error so I decided to recreate the test user manually using the following commands in the EMS:

Don’t try this at home! No, of course you can 🙂 just don’t forget to modify the variables marked below:

$exchangeServer = Get-ExchangeServer EX2010
$adSiteGuidLeft13 = $exchangeServer.Site.ObjectGuid.ToString().Replace("-","").Substring(0, 13);
$UserName = "extest_" + $adSiteGuidLeft13;
$SamAccountName = "extest_" + $adSiteGuidLeft13;
$UserPrincipalName = $SamAccountName + "@" + $exchangeserver.Domain
$mailboxdatabasename = (get-mailboxdatabase *DB01).name

#The following command will prompt to provide a password that meets the complexity requirements of your domain
New-Mailbox -Name:$UserName -Alias:$UserName -UserPrincipalName:$UserPrincipalName -SamAccountName:$SamAccountName -Database:$mailboxDatabaseName

Set-Mailbox $Username -MaxSendSize:1000KB -MaxReceiveSize:10
00KB -IssueWarningQuota:unlimited -ProhibitSendQuota:1000KB -ProhibitSendReceiveQuota:unlimited -HiddenFromAddressListsE
nabled:$True

Set-User $Username -RemotePowerShellEnabled:$true

After this, run the diagnostic command again which should now be succesful:

Test-OwaConnectivity -TestType:External -MonitoringContext:$true -TrustAnySSLCertificate:$true -LightMode:$true

Another poor Managed Availability victim saved from failure and a SCOM administrator eased!

 

 

Powershell Script to automatically sign certificate request with Internal CA and using Webserver Template

Hi,

I regularly have to sign certificate requests with the customer’s internal CA and I got lazy to always look for all the details to do it manually so I decided to automate it a little bit.

I’m not a scripting guru, far from it so feel free to improve it (and share your improvements?).

FYI:

  • The script works best if there is only one Webserver template.
  • The script will prompt you to confirm the CA
  • The script will prompt you for the location of the request and at the end, prompt you again to save the signed certificate

 

# This script is used to sign a certificate request with the internal root CA web template

# The script will look for the root CA and the web template and sign the request thereby prompting to save the signed certificate.

# Author: Jozef Wu


#Ask for the location of the request:

$ReqLocation = Read-Host "Please provide the full filepath to your cert request"



#Getting information from domain (please note that CERTUTIL will prompt to select CA, just click OK):

certutil -config - -ping | out-file $env:TEMP\CA.txt

$CA = (get-content $env:TEMP\CA.txt)[0]

$Split = $CA.IndexOf("\")

$CAhost = $CA.Substring(0, $Split)

$CATemplate = Invoke-Command -ComputerName $CAhost -ScriptBlock { Get-CATemplate | Where Name -Like *web* | Select Name | foreach { $_.Name }}

$attrib = "CertificateTemplate:" + $CATemplate

Remove-Item $env:TEMP\CA.txt


#If only one WebServer template is found, proceed to sign otherwise exit

If (($CATemplate).Count -eq 1)

{

    Write-Host "Selected Certificate Template is $CATemplate, continue? (y/n)"

    $Response = Read-Host

    If ( $response -eq "y" ) {

    certreq –config $CA –submit –attrib $attrib $ReqLocation

    }

   

    Else {

    Write-Host "Aborting Script"

    Exit

    }

}

Else

{

    Write-Host "More than one WebServer template found, please select manually. Script aborts here"

    Exit

}

 

Pop probe PopSelfTestProbe is failing on server

SSL negotiation did not complete successfully: InvalidToken

Hi,

The SCOM monitor was doing his (or her?) job well and reported the following error:

Pop probe PopSelfTestProbe is failing on server EX2016 

Pop is failing to respond to the probe on Connection or Capability requests on EX2016.



Last failed result:

Exception: System.InvalidOperationException: Unable to initialise TCP Network connection ---> System.InvalidOperationException: 

SSL negotiation did not complete successfully: InvalidToken
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Common.TcpConnection.NegotiateSSL()
at Microsoft.Exchange.Monitoring.ActiveMonitoring.PopImap.Probes.PopImapProbeUtil.CreatePopSSLStateObject(IPEndPoint targetAddress, ProbeResult result)
--- 

End of inner exception stack trace ---
at Microsoft.Exchange.Monitoring.ActiveMonitoring.PopImap.Probes.PopSelfTestProbe.HandleSocketError(Exception e)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.PopImap.Probes.PopImapProbeBase.HandleConnectionExceptions(PopImapProbeStateObject probeTrackingObject)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.PopImap.Probes.PopSelfTestProbe.DoWork(CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<ExecuteAsync>d__b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<StartExecutingAsync>d__7.MoveNext()




Diagnostic command: Invoke-MonitoringProbe -Identity:"POP.Protocol\PopSelfTestProbe" -Server:EX2016 | fl

Sometimes, rerunning the probe (the Diagnostic command at the end of the SCOM error above) can solve the issue if it was a transient error but in this case it wasn’t.

My eye fell on the following line “SSL negotiation did not complete successfully: InvalidToken” and it led me to inspect the certificates. I noticed that the POP service was assigned to an internal CA signed certificate which did not include the Exchange server FQDN.

Adding POP to the Exchange server self-signed certificate was not possible because of the following error:

WARNING: This certificate with thumbprint 2B70359DC0BE9CAD58F965A3B523 and subject ‘COMPANYEXCH02’ cannot used for POP SSL/TLS connections because the subject is not a Fully Qualified Domain Name (FQDN). Use command Set-POPSettings to set X509CertificateName to the FQDN of the service.

It doesn’t happen often that an error tells you exactly how to solve a problem but here it was:

Set-POPSettings -X509CertificateName EX2016

Now I restarted the POP services (including backend) and…

Problem solved!

 

Welcome

Hi,

This is my Exchange & Office 365 blog where I share my insights, solutions and general experiences as an IT consultant.

I don’t consider myself to be a guru or anything like it but I still hope my ramblings can help any significant other out there who also goes through the daily struggle with Exchange and Office 365.

Feel free to comment or correct. We can all learn from each other.

Thanks!

 

Jozef Woo (aka nUber)