Lockdown Diary – Metered Internet Connections and Broken ConfigMgr Clients

May 23, 2020

Do you know how hard it is to find things when you don’t know what to search for??

I’m paranoid. We recently upgraded our site to ConfigMgr 2002, first fast ring, then the hotfix. We also installed a Cloud Management Gateway, re-worked our Boundary Groups to handle VPN better, added a second Management Point and generally made a butt-load of changes to our environment to help our remote clients have access to content more easily. I’ve been frantically spot checking clients to ensure they are working now and hoping that one of the many changes hasn’t caused them to drop off the face of the earth. There was also a bug that got fixed in slow ring/hotfix, related to client failing to authenticate the CMG that wanted to ensure was working now. Unfortunately, when you go looking into client logs, you can end up chasing lots of rabbits. On of the issues that I’m having is that some of our clients that are likely at the office are just going to be offline until everyone returns to work post-COVID-19. At the same time, we have some other number of online clients that are showing no client or no recent heartbeats but when I ping them, a decent number respond. On a regular basis, we have about 300 machines in some sort of unknown state and we have just learned to live with it since it’s largely environmental. We have the PFE client health tool implemented, but honestly, I’m not sure if it’s helping or hurting at this point.

I feel like I should stat the purpose here - I want to disable metering and get ConfigMgr clients to check back in without having to reboot the client.

DISCLAIMER: The ‘solution’ provided below uses several ‘unsupported’ steps to achieve the intended results and is provided for informational purposes only.

Without further ado, here’s the brain dump firehose…

See, what had happened was…

I used Recast Right Click tools (free community edition) not a sponsored post :-) to trigger a client repair on around 500 devices. About 60 of the machines were online even though they were showing no client in the console. I waited to see if they would check in and they never did. Started looking at client logs and noticed that the repairs seemed to be working and the clients appeared to be fine, but when I checked CcmMessaging.log, it was full of these errors:

1
2
3

Request to http://CM01.ASD.net/ccm_system/request cannot be fulfilled since use of metered network is not allowed.

Post to https://CM01.ASD.net/ccm_system_windowsauth/request failed with 0x87d00231.

I opened a remote PowerShell console to some of the machines and I started checking to see if metering had been enabled on the wireless adapters by the end user. I ran some NETSH commands to find the profiles where metering was enabled and the clients began working. I was excited until I got to my next machine and discovered that it was on the LAN at the office and we don’t enable metering on any of our adapters by default.

$ProfileList = Get-NetConnectionProfile
ForEach ($Profile in $ProfileList.Name) {
    (netsh wlan show profile name="$($Profile)")
}

I tried the same NETSH commands and they didn’t work on the LAN adapters. It took me a while but I stumbled on several posts that had bits and pieces that helped find what I needed. There’s a fair amount of detail upcoming, but the script is at the end. I want to include all of the components that I discovered so I can consolidate everything into one place.

Also, I’m just making most of this up based on what I have been able to piece together. I have spent WAY too much time digging and just need to move on. I could be REALLY wrong, but it’s the best I’ve got. If you know more, send me a note and I’ll delete it, I mean, I’ll update the post.

Network Adapter Metering

When setting up your network adapters in Windows 10, I’m sure you’ve seen this section under the properties for your network adapters. By enabling metering, you tell Windows to limit the amount of data that it and other applications can put through the adapter. This is a way to reduce consumption of pay-per-use internet plans and or wifi hotspots, etc.

One problem with this setting is that it will actually disable Windows updates (there’s an option to override this as well) AND more importantly, it will prevent the installation of the ConfigMgr client. Also, if you haven’t changed your Client Settings policy from the default ‘Block’, devices with the client installed won’t communicate with your MP - as shown in the error log above.

Another problem is that there appear to be Group Policies for disabling the ability for users to change this setting for wireless adapters or cellular cards, but I wasn’t able to find one that prevented users from setting metering on their wired LAN adapters.

It also appears that there have been changes to where/how you manage metering in the registry/Group Policy in later versions of Windows (1709 and later).

Computer Configuration \ Administrative Templates \ Network \ WLAN Service \ WLAN Media Cost

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\NetworkList\DefaultMediaCost

Basically, there are some options that are somewhat difficult to lock down that users can change and break things.

ConfigMgr Client Event Handling

The ConfigMgr client has lots of event handlers that it uses to monitor system processes and trigger processes in response to the events. One of the items it monitors is network connectivity and adapter events. Whenever you enable metering on your adapter, you can watch the CcmNotificationAgent.log to see that the network cost was changed to 2 (Enabled/Fixed/Metered) or if you disable, it will show 1 (Disabled/Unlimited/Not Metered). If you enable Debug logging on your ConfigMgr client, you will see even more info (not shown). Martin has a great script to enable debug logging here.

Cost Network state has been updated to 2, trigger client endpoint reset connection according to the change.
Cancel the active connector here, tcp connection will be disconnected immediately while http connection will be timeout evetually.
Connection is reset
Critical Battery: [FALSE]
Connection Standy: [FALSE]
Network allowed to use: [FALSE]
Sleep 59 seconds to restart client...

Cost Network state has been updated to 1, trigger client endpoint reset connection according to the change.
Critical Battery: [FALSE]
Connection Standy: [FALSE]
Network allowed to use: [True]

You can even run a PowerShell command to check the ConfigMgr client for it’s current Network Cost - it will return 1 or 2. From what I can tell, ConfigMgr monitors for a State Change event on the network adapter, then updates it’s network cost value and caches it. As far as I could determine, there doesn’t appear to be a way to force the client to check for the network cost again nor is there a way to trigger a network state change that it would see.

`1`	`Invoke-CimMethod -Namespace "root\ccm\ClientSDK" -ClassName "CCM_ClientUtilities" -MethodName GetNetworkCost`

You can watch the NetworkProfile event log and see the State Change event when you enable/disable metering on the adapter.

Log Name:      Microsoft-Windows-NetworkProfile/Operational
Source:        Microsoft-Windows-NetworkProfile
Date:          5/22/2020 2:15:58 PM
Event ID:      4004
Task Category: None
Level:         Information
Keywords:      
User:          LOCAL SERVICE
Computer:     mycomputer.asd.net 
Description:
Network State Change Fired
	New Internet Connection Profile: false
	Connection Cost Changed: true
	Domain Connectivity Level Changed: false
	Network Connectivity Level Changed: false
	Host Name Changed: false
	Wwan Registration State Changed: false
	Tethering Operational State Changed: false
	Tethering Client Count Changed: false
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-NetworkProfile" Guid="{fbcfac3f-8459-419f-8e48-1f0b49cdb85e}" />
    <EventID>4004</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime="2020-05-22T19:15:58.412838100Z" />
    <EventRecordID>113</EventRecordID>
    <Correlation />
    <Execution ProcessID="1984" ThreadID="1308" />
    <Channel>Microsoft-Windows-NetworkProfile/Operational</Channel>
    <Computer>mycomputer.asd.net</Computer>
    <Security UserID="S-1-5-19" />
  </System>
  <EventData>
    <Data Name="NewInternetConnectionProfile">false</Data>
    <Data Name="ConnectionCostChanged">true</Data>
    <Data Name="DomainConnectivityLevelChanged">false</Data>
    <Data Name="NetworkConnectivityLevelChanged">false</Data>
    <Data Name="HostNameChanged">false</Data>
    <Data Name="WwanRegistrationStateChanged">false</Data>
    <Data Name="TetheringOperationalStateChanged">false</Data>
    <Data Name="TetheringClientCountChanged">false</Data>
  </EventData>
</Event>

You can see the ConnectionCostChanged entry. This is what ConfigMgr is looking for.

Here’s what ccmsetup.log looks like if you have metering enabled and try to install the ConfigMgr client. It’s a hardcoded check too, so no overriding it.

Failed to connect to machine policy namespace. 0x8004100e
Client deployment cannot be fulfilled since use of metered network is not allowed.
Failed to parse '"C:\WINDOWS\ccmsetup\ccmsetup.exe" /runservice /ForceInstall /ignoreskipupgrade /config:MobileClient.tcf' with error 0x87d00227
Sending state '329'...
Updating MDM_ConfigSetting.ClientDeploymentErrorCode with value 2278556199
Failed to get client version for sending state messages. Error 0x8004100e
[] Params to send '5.0.8968.1014 Deployment "C:\WINDOWS\ccmsetup\ccmsetup.exe" /runservice /ForceInstall /ignoreskipupgrade /config:MobileClient.tcf'
Sending Fallback Status Point message to 'cm01.asd.net', STATEID='329'.
<ClientDeploymentMessage ErrorCode="-2016411097"><Client Baseline="1" BaselineCookie="" Platform="2" Langs=""/></ClientDeploymentMessage>
State message with TopicType 800 and TopicId {732AC9D5-1BE6-4EDC-9488-5AE42B63CA5D} has been sent to the FSP
Failed to connect to policy namespace. Error 0x8004100e
Sending state '301'...
Updating MDM_ConfigSetting.ClientDeploymentErrorCode with value 2278556199
CcmSetup failed with error code 0x87d00227

The Windows Network Adapter Black Hole

I’ll be the first to admit, I don’t like networking. I dislike it even more after this adventure. So we’ve seen where in the UI you can enable/disable metering. We’ve seen in GPO where you can configure WLAN cost options. We’ve seen the network event that gets triggered when you make a change. What we haven’t seen is how you can programatically change costing on a LAN adapter in a way that causes the ConfigMgr to see the event and trigger a re-cache of costing for the network adapter. Spoiler - I can’t find a way, but here’s how far I got.

Data Usage

There’s a Windows system DLL dusmapi.dll(Data Usage API) and a service dusmsvc (Data Usage) that are used to manage your network adapter profile settings related to - you guessed it - DATA USAGE!!

By watching ProcMon, I could see that dusmapi.dll was being called but I can’t figure out how to import the dll into PowerShell or VS to try to manually call it. However, I did find find a dump of all of the functions in the dll.

DusmEnumConnectionList
DusmEnumProfileList
DusmFlushCostCache
DusmFree
DusmGetAttributedNetworkUsage
DusmGetConnectionListNetworkUsage
DusmGetNetworkUsage
DusmGetProviderNetworkUsage
DusmQueryBackgroundRestriction
DusmQueryConnectionProperties
DusmQueryCost
DusmQueryDataPlan
DusmQueryGlobalDpuState
DusmQueryOperatorCost
DusmQueryOperatorDataPlan
DusmQuerySource
DusmQueryUserCost
DusmQueryUserDataPlan
DusmResetNetworkUsage
DusmSetAttributionMapping
DusmSetBackgroundRestriction
DusmSetOperatorCost
DusmSetOperatorDataPlan
DusmSetSource
DusmSetUserCost
DusmSetUserDataPlan

On line 25 we can see that there’s a function called DusmSetUserCost. That’s the one we want. I’ve searched GitHub, in the Windows networking PowerShell module and everywhere else and I can’t find any info on how to call this but I’m certain it’s the key to triggering the costing state change event on the adapter.

The other related bit here is that the setting that gets created by toggling metering on an adapter is a registry key Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DusmSvc\Profiles\<AdapterGUID>\*\UserCost. The value will be set to 2 if it’s enabled and 0 if you disable it. If you never touch it, you will only have the base key of Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DusmSvc\Profiles.

A light in the darkness

There are other references to network costs, but only for querying them, nothing for setting them.

Run this in PowerShell and you can dump out the cost info for all of your network adapter profiles.

[void][Windows.Networking.Connectivity.NetworkInformation, Windows, ContentType = WindowsRuntime]
Windows.Networking.Connectivity.NetworkInformation]::GetInternetConnectionProfile().GetConnectionCost()


output:
PS C:\> [void][Windows.Networking.Connectivity.NetworkInformation, Windows, ContentType = WindowsRuntime]
[Windows.Networking.Connectivity.NetworkInformation]::GetInternetConnectionProfile().GetConnectionCost()


ApproachingDataLimit          : False
NetworkCostType               : Unrestricted
OverDataLimit                 : False
Roaming                       : False
BackgroundDataUsageRestricted : False

A nice guy named Wil (with 1 L) Taylor even has a module to help dump cost info out. Unfortunately, there’s not a module for modifying the settings.

I have to give credit to Franck Richard for his nice write ups on this topic as well. He provided some of the only pieces to this puzzle that I found initially. These 2 posts talked about the issues I was facing and he even has a script for it editing it that I used bits of in my final solution.
http://franckrichard.blogspot.com/2018/11/sccm-client-certificate-value-set-to.html http://franckrichard.blogspot.com/2018/11/set-onoff-metered-ethernet-connection.html

The Script

Here’s a link to my GitHub repo where you can download the latest version of the script.

The script has 3 main parts. LAN, WLAN and ConfigMgr. It checks the registry for any LAN profiles set using the Settings app in Windows. These will be under the dusmsvc registry key here: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DusmSvc\Profiles. Then it uses NETSH to find WLAN profiles with metering not set to Unlimited. And finally it checks to see of the ConfigMgr client costing is set to metered. Examples and info are included in the script.

Alternatives

As soon as I hit publish, I considered that perhaps I could have handled this slightly differently. Honestly, I wrote all of the code before I found the ConfigMgr CCM_NetworkSettings class. Now that I’m thinking about it and understand things more, here’s how I would ACTUALLY do it.

This is part of the final code in the repo. It’s the part that resets the CCM_NetworkSettings class. The cool part here is that it allows the client to check in and get new policies, even if metering is enabled. My issue was that we had our client settings policy set to Block. If I remove the policy (with the code below), the clients can get an updates.

$CCMNetworkCost = (Invoke-CimMethod -Namespace "root\ccm\ClientSDK" -ClassName "CCM_ClientUtilities" -MethodName GetNetworkCost).Value
Write-Host "ConfigMgr Cost: $($CCMNetworkCost)"

If($CCMNetworkCost -ne 1) {
    #Set metering to 1, restart client so it will check in, remove the policy instance, then get new policies
    $PolicyNameSpace = "root\ccm\Policy\Machine\ActualConfig"
    $NwClassName = "CCM_NetworkSettings"
    $obj = Get-CIMInstance -Namespace $PolicyNameSpace -ClassName $NwClassName
    If($obj.MeteredNetworkUsage -ne 1) {
        Write-Host "ConfigMgr MeteredNetworkUsage is set to $($obj.MeteredNetworkUsage)"
        Write-Host "Reseting ConfigMgr CCM_NetworkSettings Policy"
        #Set usage to 1 in the policy first. This allows the client to go get policies.        #We will delete the entry at the end to ensure that the setting gets re-applied after a policy refresh.
        #In testing, policies didn't reapply without removing the entry.
        $obj | Set-CimInstance -Property @{MeteredNetworkUsage=1}  
        Restart-Service -Name ccmexec -ErrorAction SilentlyContinue
        #Give policies time to churn
        Start-Sleep -Seconds 30 
        #Remove the policy entry from WMI
        $obj | Remove-CimInstance
        Invoke-CimMethod -Namespace "root\ccm" -ClassName "SMS_Client" -MethodName RequestMachinePolicy -Arguments @{uFlags = [uint32]1 }
        Invoke-CimMethod -Namespace "root\ccm" -ClassName "SMS_Client" -MethodName EvaluateMachinePolicy
    }
}

Next, I would add a Configuration Item or a scheduled script to check for and remove metering. I haven’t test Group Policy preferences but since the registry key has a * in the path (who decided that was a good idea??) and * is a wildcard pretty much EVERYWHERE, I’m not sure how well that will work. Either way, having a mechanism to block or remove this from being set again would be the next step.

And instead of a script, I think I could have just updated the client settings policy to Allow ConfigMgr over metered connections then used Right Click Tools to reset client policies on any devices that I suspected had metering enabled.

The End

This has been a fun dive into the bowels of network connectivity. I’m REALLY hoping someone sends me an email with the code to update costs using the Windows API. Feels like there’s still some opportunity for polishing this turd.