Autopilot Profile causes Device Rename after ConfigMgr OSD Task Sequence and Breaks AD Domain Trust

We got some new hardware models in this week and added drivers to our ConfigMgr OSD Task Sequence (with Windows 10 1909 serviced with November 2020 updates) to test. One of the devices kept ending up with a broken domain trust relationship when you attempt to log in immediately following build completion.

The security database on the server does not have a computer account for this workstation trust relationship
The security database on the server does not have a computer account for this workstation trust relationship

One clue that we had was that something was renaming the device. We have a very basic naming convention SITE-SERIAL where SITE is 2-5 character code for the AD site/OU a device will be assigned to. The device was be in Active Directory in the right OU and properly domain joined, but at the logon screen in the username field if we typed .\ to show the device name instead of the domain name, the device name was only the SERIAL, no SITE- prefix. The smsts.log, setupact.log, unattend.xml all showed the correct device name through the entire process. Here’s the rest of the story…

Note - the serial in the screenshots doesn’t match the logs due to obfuscation of prod data
Note - the serial in the screenshots doesn't match the logs due to obfuscation of prod data

Trudging Through Logs

Like most people, I love (or is is loathe) digging through logs looking for a needle in a haystack. In this case, I needed to find evidence that showed that the device was properly named, but then was changed sometime later. I knew that it was named correctly at least until the domain join step and after digging through smsts.log, it was clear that it wasn’t being changed during or by the Task Sequence at all. It was coming afterwards. I moved to the Event Viewer and started looking for Event ID 6011 which would show that the device name had been changed and the device had been rebooted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Information	12/1/2020 2:22:35 PM	EventLog	6011	None
Log Name:      System
Source:        EventLog
Date:          12/1/2020 2:22:35 PM
Event ID:      6011
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      12345678.asd.net
Description:
The NetBIOS name and DNS host name of this machine have been changed from ASD-12345678 to 12345678.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="EventLog" />
    <EventID Qualifiers="32768">6011</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2020-12-01T20:22:35.941883400Z" />
    <EventRecordID>927</EventRecordID>
    <Channel>System</Channel>
    <Computer>12345678.asd.net</Computer>
    <Security />
  </System>
  <EventData>
    <Data>ASD-12345678</Data>
    <Data>12345678</Data>
  </EventData>
</Event>

Once I had the base timestamp to work from, I was able to narrow my search to a specific time that this could have happened. I scrolled back in time and and found Event ID 1074 which showed that the reboot was triggered by CloudExperienceHost.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
Information	12/1/2020 2:22:04 PM	User32	1074	None
Log Name:      System
Source:        User32
Date:          12/1/2020 2:22:04 PM
Event ID:      1074
Task Category: None
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      ASD-12345678.asd.net
Description:
The process C:\Windows\System32\CloudExperienceHostBroker.exe (ASD-12345678) has initiated the restart of computer ASD-12345678 on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Reconfiguration (Unplanned)
 Reason Code: 0x20004
 Shutdown Type: restart
 Comment: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="User32" Guid="{b0aa8734-56f7-41cc-b2f4-de228e98b946}" EventSourceName="User32" />
    <EventID Qualifiers="32768">1074</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2020-12-01T20:22:04.988802400Z" />
    <EventRecordID>918</EventRecordID>
    <Correlation />
    <Execution ProcessID="836" ThreadID="3048" />
    <Channel>System</Channel>
    <Computer>ASD-12345678.asd.net</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="param1">C:\Windows\System32\CloudExperienceHostBroker.exe (ASD-12345678)</Data>
    <Data Name="param2">ASD-12345678</Data>
    <Data Name="param3">Operating System: Reconfiguration (Unplanned)</Data>
    <Data Name="param4">0x20004</Data>
    <Data Name="param5">restart</Data>
    <Data Name="param6">
    </Data>
    <Data Name="param7">NT AUTHORITY\SYSTEM</Data>
  </EventData>
</Event>

When I looked at C:\Windows\Panther\UnattendGC\setupact.log I noticed that CloudExperienceHost had failed and created a memory dump around the same time the device rename and matched up with the timestamp on the restart event.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
2020-12-01 14:21:45, Info                         [taskhostw.exe] OOBE Monitor event received: 103
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] Failed to load oobe.xml file at C:\WINDOWS\system32\oobe\info\oobe.xml with hr=0x80070003
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] Failed to load oobe.xml file at C:\WINDOWS\system32\oobe\info\default\oobe.xml with hr=0x80070003
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] Failed to load oobe.xml file at C:\WINDOWS\system32\oobe\info\244\oobe.xml with hr=0x80070003
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] Failed to load oobe.xml file at C:\WINDOWS\system32\oobe\info\default\1033\oobe.xml with hr=0x80070003
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] Failed to load oobe.xml file at C:\WINDOWS\system32\oobe\info\244\1033\oobe.xml with hr=0x80070003
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] Opting in to IncludeDriverSets
2020-12-01 14:21:50, Info                         [CloudExperienceHostBroker.exe] CZDPTask setting OOBE_GLOBAL_SETTING_ZDPSTATUS ZDP_Scanning.
2020-12-01 14:21:51, Info                         [taskhostw.exe] OOBE Monitor event received: 104
2020-12-01 14:21:56, Info                         [CloudExperienceHostBroker.exe] ISearchCompletedCallback::Invoke called
2020-12-01 14:21:56, Info                         [CloudExperienceHostBroker.exe] Wait result from WU ZDP Search: 0X1
2020-12-01 14:21:56, Info                         [CloudExperienceHostBroker.exe] Found 0 updates in collection.
2020-12-01 14:21:56, Info                         [CloudExperienceHostBroker.exe] CZDPTask::CancelTask() Invoked.
2020-12-01 14:21:56, Info                         [CloudExperienceHostBroker.exe] CZDPTask setting OOBE_GLOBAL_SETTING_ZDPSTATUS ZDP_Finished.
2020-12-01 14:22:09, Info                         [taskhostw.exe] OOBE Monitor task stopped
2020-12-01 14:22:36, Info                         [svchost.exe] Enter WinReIsWimBootEnabled
2020-12-01 14:22:36, Info                         [svchost.exe] Parameters: pszWinDir: NULL
2020-12-01 14:22:36, Info                         [svchost.exe] Exit WinReIsWimBootEnabled returns 0 with last error: 0x0
2020-12-01 14:22:38, Info                         [taskhostw.exe] OOBE Monitor task background thread started
2020-12-01 14:22:39, Info                         [taskhostw.exe] OOBE Monitor event received: 101
2020-12-01 14:22:39, Info                         [taskhostw.exe] OOBE Monitor event received: 102
2020-12-01 14:22:40, Info                         [taskhostw.exe] OOBE Monitor event received: 103
2020-12-01 14:22:42, Info                         [CloudExperienceHostBroker.exe] UserOOBEController::Exit() started [1]
2020-12-01 14:22:42, Info                         [taskhostw.exe] OOBE Monitor event received: 999
2020-12-01 14:22:42, Info                         [CloudExperienceHostBroker.exe] Clearing Auto-logon values per request
2020-12-01 14:22:42, Error                        [CloudExperienceHostBroker.exe] Disabling default account failed [hr=0xD00000E5]
2020-12-01 14:25:40, FatalError [0x090001] PANTHR Exception (code 0xC0000005: ACCESS_VIOLATION) occurred at 0x00007FFDDADD2236 in C:\Windows\System32\oobe\UserOOBE.dll (+0000000000022236).  Minidump attached (47855 bytes) to diagerr.xml and C:\WINDOWS\Panther\UnattendGC\mndF37E.diagerr.mdmp.

For no real reason, I decided to check if the machine was getting an Autopilot profile. The Event Viewer showed that the device had gotten an Autopilot json file, which seemed odd since this device shouldn’t have had a profile assigned yet. I opened C:\Windows\ServiceState\Autopilot and found AutopilotDDSZTDFile.json. In here, I found the smoking gun!

A Square Dozen Image

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
{
    "AutopilotServiceCorrelationId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "ZtdRegistrationId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "AadDeviceId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "CloudAssignedOobeConfig": 284,
    "CloudAssignedDomainJoinMethod": 0,
    "CloudAssignedForcedEnrollment": 1,
    "CloudAssignedTenantDomain": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "CloudAssignedTenantId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "CloudAssignedMdmId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "CloudAssignedRegion": "os-default",
    "CloudAssignedLanguage": "os-default",
    "CloudAssignedDeviceName": "%SERIAL%",
    "DeploymentProfileName": "Autopilot",
    "IsExplicitProfileAssignment": true,
    "CloudAssignedAutopilotUpdateDisabled": 1,
    "CloudAssignedPrivacyDiagnostics": 0,
    "HybridJoinSkipDCConnectivityCheck": 0,
    "CloudAssignedAutopilotUpdateTimeout": 1800000,
    "PolicyDownloadDate": "2020-12-01T20:21:49Z"
}

Making Sense of Things

As you may know, you can only assign custom naming to Azure AD Joined devices, not Hybrid, during Autopilot. How could this be happening during OSD?

During OSD, the device is installing Windows then apps and settings and is locked out until it’s done. Once it is finished, it will reboot and is supposed to skip OOBE and go to the logon screen. We recently removed our custom Unattend.XML file from our TS since we were specifically using it to suppress OOBE post OSD using some now deprecated settings (SkipUserOOBE & SkipMachineOOBE) that still work.

A Square Dozen Image
https://docs.microsoft.com/windows/deployment/deploy-windows-mdt/create-a-windows-10-reference-image#edit-the-unattendxml-file-for-windows10-enterprise

A Square Dozen Image
Network OOBE Screen

So it appears in the brief moment where OOBE is being shown, the device is contacting the Autopilot service and pulling down an assigned Autopilot profile.

Finding a Workaround

This next section describes the troubleshooting process that I used to validate the source of the issue.

Option 1 (supported)

The ‘supported’ solution to this problem would be to NOT assign Autopilot profiles (specifically ones with naming templates configured) to devices that you are NOT Autopiloting. This is the cleanest option and doesn’t require adding in deprecated configurations to your unattend.xml. This SHOULDN’T be happening, but as Michael Niehaus pointed out to me after reading this post, he tried for 2 years to get this fixed and ultimately, just don’t do it (assign an Autopilot profile to a non-Autopilot device) that way.

Option 2 (unsupported)

To test my theory about OOBE, I added unattend.XML back to the Task Sequence and built the machine. The device didn’t pull down the Autopilot profile and didn’t get renamed. I then removed the deprecated lines from the unattend.xml file and built the machine again. This time it failed the same as before, confirming my suspicion.

This is my default unattend.xml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
<?xml version="1.0"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="oobeSystem">
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <OOBE>
                <SkipUserOOBE>true</SkipUserOOBE>
                <SkipMachineOOBE>true</SkipMachineOOBE>
                <NetworkLocation>Work</NetworkLocation>
                <ProtectYourPC>3</ProtectYourPC>
            </OOBE>
            <TimeZone>Central Standard Time</TimeZone>
        </component>
        <component name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>0409:00000409</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
    </settings>
    <settings pass="specialize">
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <ComputerName>ComputerName</ComputerName>
        </component>
    </settings>
    <cpi:offlineImage cpi:source="wim:c:/temp/install.wim#Windows 10 Enterprise" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

Apply Operating System with unattend.xml
Apply Operating System with unattend.xml

Here’s the unattend.xml after the Task Sequence Apply steps have completed just before the Setup Windows and ConfigMgr step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
<?xml version="1.0"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
	<cpi:offlineImage cpi:source="wim:c:/temp/install.wim#Windows 10 Enterprise" xmlns:cpi="urn:schemas-microsoft-com:cpi"/>
	<settings xmlns="urn:schemas-microsoft-com:unattend" pass="oobeSystem"><component name="Microsoft-Windows-Shell-Setup" language="neutral" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
			<OOBE>
			    <SkipUserOOBE>true</SkipUserOOBE>
                <SkipMachineOOBE>true</SkipMachineOOBE>
				<ProtectYourPC>3</ProtectYourPC>
				<NetworkLocation>Work</NetworkLocation>
				<HideEULAPage>true</HideEULAPage>
			</OOBE>
			<TimeZone>Central Standard Time</TimeZone>
			<RegisteredOrganization>A Square Dozen Lab</RegisteredOrganization>
			<UserAccounts>
				<AdministratorPassword>
					<Value>XXXXXXXXXXXXXXXXX</Value>
					<PlainText>false</PlainText>
				</AdministratorPassword>
			</UserAccounts>
			<RegisteredOwner>A Square Dozen Lab</RegisteredOwner>
		</component>
		<component name="Microsoft-Windows-International-Core" language="neutral" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
			<SystemLocale>en-US</SystemLocale>
			<UserLocale>en-US</UserLocale>
			<UILanguage>en-US</UILanguage>
			<InputLocale>0409:00000409</InputLocale>
			<UILanguageFallback>en-US</UILanguageFallback>
		</component>
	</settings><settings xmlns="urn:schemas-microsoft-com:unattend" pass="specialize"><component name="Microsoft-Windows-Shell-Setup" language="neutral" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
			<ComputerName>ASD-123245678</ComputerName>
		</component>
		<component name="Microsoft-Windows-UnattendedJoin" language="neutral" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
			<Identification>
				<Credentials>
					<Username>ASDAdmin</Username>
					<Domain>ASD</Domain>
					<Password>XXXXXXXXXX</Password>
				</Credentials>
				<MachineObjectOU>OU=Workstations,OU=Lab,DC=asd,DC=net</MachineObjectOU>
				<JoinDomain>ASD.NET</JoinDomain>
			</Identification>
		</component>
		<component name="Microsoft-Windows-Deployment" language="neutral" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
			<RunSynchronous>
				<RunSynchronousCommand><Order>1</Order>
					<Description>disable user account page</Description>
					<Path>reg add HKLM\Software\Microsoft\Windows\CurrentVersion\Setup\OOBE /v UnattendCreatedUser /t REG_DWORD /d 1 /f</Path>
				</RunSynchronousCommand>
			</RunSynchronous>
		</component>
	</settings></unattend>

As you can see on lines 6 & 7 above SkipUserOOBE and SkipMachineOOBE are included in the XML and make the issue go away.

Wrapping it all up

This was a weird one for sure and this feels like a bug. The workarounds are simple enough that this isn’t a show stopped, but trying to find the source of the bug was definitely a head scratcher. I haven’t had time to test with any other OS versions or even non-serviced versions yet but would be interested to see if they suffer the same issues. I seem to recall that the OOBE issues that required the custom unattend.xml only showed up on offline serviced or captured images, but not on OEM media.

P.S.

You may be asking yourself why I was doing OSD when I had AutoPilot assigned for AAD Join. Well, the device we were testing was a demo unit that had been registered to another company’s tenant and had THEIR Autopilot profile assigned. I had no way of deleting the device or unassigning the profile. The vendor has since resolved the issue, but it helped highlight a deficiency in their process as well, so win-win I guess.

Also, I was unable to find any Autopilot logs that actually show the device rename being triggered which also bugs me…