SCCM Collection Evaluation – Troubleshooting Incremental Update Issues

February 21, 2019

About a month ago, I wrote a post for System Center Dudes related to the changes to recurring collection schedules in SCCM 1810. In our environment, we removed the default recurring schedules from about 1750 device collections that use Direct membership rules. Up until that point, we had seen some strange collection behaviors, but never at the level that we began to experience them post ConfigMgr 1810. (Let me stop here and say that 1810 didn’t break anything. I have confirmed that the behavior that I will describe exists in 1802 as well.).

In the past month, we started to see issues with newly built devices not populating collections even though the Direct membership rules were being created and the devices eventually populated the limiting collections. The only way devices would populate collections was by manually refreshing the collection or adding/removing a rule which would triggers a collection update. I opened a case with Premier Support and worked with Vinay Pinmani, who helped investigate. So, here’s what I know about SCCM Device collections now:

‘Broken’ Collection Structure

For our production collections, we have an All Workstations (Parent in the example below) collection limited to All Systems. It is used as the limiting collection for all of our workstation application deployment collections (Child 2 in the example below). For testing, we added a few additional collections to compare the experience.

All Systems
- Limiting Collection: None
- Schedule: 4:00 AM daily (default)
- Incremental Refresh: Enabled - 5 min refresh cycle (default)
Parent - Incremental
- Limiting Collection: All Systems
- Schedule: None
- Incremental Refresh: Enabled
- Rules: Query Rule to only return devices named %test% * Note: in production this query gets all workstations but we limited the members for testing to prevent unwanted updates from propagating to the parent and skewing testing.
Child 1 - Incremental
- Limiting Collection: Parent - Incremental
- Schedule: None
- Incremental Refresh Enabled
- Rules: None
Child 2 - Direct Only
- Limiting Collection: Parent - Incremental
- Schedule: None
- Incremental Refresh: Disabled
- Rules: None
Child 3 - Scheduled
- Limiting Collection: Parent - Incremental
- Schedule: Daily 11:21 AM
- Incremental Refresh: Disabled
- Rules: None

Reproducing the Issue

During Operating System Deployment (OSD), we have a collection mapping process where we get a list of collections from a Source computer and add direct membership rules for the new device. This is where we were seeing our issues. Initially, I thought it may be an issue with our REST API calling the wrong WMI commands to create collection Direct membership rules. Upon further testing, I was able to reproduce the issue by manually importing a new device from the console (such a relief knowing it wasn’t my code!).

Test Scenario 1 - Import New Device. Manually Add to Collections

In the ConfigMgr Console import a new device named Test1 accepting all defaults.

Wait for the device populate the Parent collection (Default Incremental update is 5 mins), then manually add Direct rules to each of the Child collections by selecting the collection and clicking Add Resources. This action cause a .UDC file to be added to the colleval.box and processed. This file simply tells SCCM to trigger an update on the collection right away. It’s equivalent to clicking Update Membership on a collection. At this point, Test1 should be added to ALL Child collections.

Summary
This test worked as expected. The device made it into all collections. My guess is that MOST collection membership rules are added After a device is already in the limiting collection, so this just works because of the UDC file being added to the colleval.box.

Test Scenario 2 - Import New Device and Pre-Add to Collections

In the ConfigMgr Console import a new device named Test2. On the Choose Target Collections step, click Add then select all 3 Child collections then accept the remaining defaults.

Click Add to select collections. Add device only to child collections.

Check the child collection properties now and you will see Direct membership rules have been added to them.

Wait for the All Systems incremental again. DON’T manually update ANY collection at this stage or you will corrupt the test. You can monitor colleval.log and should see the collectionID of your limiting collection added to the Graph for processing. Once the incremental evaluation has added the new device to All Systems, check the limiting collection (Parent - Incremental) to verify that the device is now a member of it as well.

When you check the membership for the child collections you will find that only Child 1 was updated. Note the Date Modified and Membership Change Time columns (you may have to right click on the column header and enable them). Each child has a 11:15 PM Date Modified timestamp. This is the time that the new device was imported and the direct membership rules were created. The Membership Change Time on the full chain from All Systems to Child 1 were all updated as part of the incremental refresh cycle, but Child 2 & 3 did not see any membership changes.

Summary
When Child 3’s schedule finally runs, the device will populate the collection. Child 2 will never see the device Test2 unless the collection is updated either manually or by making a rule change which will trigger the UDC file and a full update.

Test Scenario 3 - Import New Device with Pre-Add to Collections and Manual All System Update

For this last scenario, you will want to monitor the colleval.log and get the timing just right. You will create a new device named Test3.

When colleval completes the next Incremental, follow the same steps as in Test 2, however this time, import the device and immediately Manually update All Systems. You will see colleval add all collections to the evaluation graph and perform a full update. You will also see that this will cause everything in the chain to perform a full update.

Check the Child collections and you will see that all 3 devices are in all collections.

Summary
By making a change on All Systems and forcing a full update between Incremental updates, we were able to force all child collections to update as well. Since it updated everything, Test2 was also added to Child 2 and 3.

The Details

All of the following assumes that you want your child collections to stay updated automatically. If you do not desire this behavior, this doesn’t really apply. After testing several more scenarios including query rules for Child collections instead of direct, here’s what we have concluded:

Incremental updates clear the ‘changed’ flag on collections as it processes them.
Full updates (scheduled or manual) check for the ‘changed’ flag on collections at each level of the dependency chain. If no changes are pending, collections at the next level are not processed.
If ANY limiting collection in the collection chain has Incremental updates enabled ALL child collections must also have Incremental enabled OR a recurring schedule OR can only have rules added when the device is already in the limiting collection.
If a limiting collection has pending changes and is updated by schedule or manually, all dependent collections (Child, Include or Exclude collections) perform full updates when the limiting collection updates.
If there are unprocessed changes on an Incremental-enabled limiting collection BETWEEN incremental updates AND the collection is refreshed (manually or by schedule), a full update will be performed on the limiting collection and all items in it’s dependency chain.
All of these scenarios also apply to removal from collections when a device is deleted or is no longer in the limiting collection.

What Now?

I can’t tell you what’s best for you. For our environment, since we have need to add devices to collections during OSD, we have removed Incremental from all limiting collections and added recurring schedules instead. We are testing performance with the changes, but currently have our All Workstations collection scheduled to refresh every 30 minutes and it takes ~12 minutes for a full update to process through all collections. Hopefully this will help things along. Also, let me be clear, do what’s best for your environment. We will still continue to use Incremental updates for our high priority collections, but we have a better understand better how they must be handled to ensure they function properly. You could also add incrementals for everything (tread carefully), but that’s totally up to you! Use colleval.exe to see how things look and go from there.

Resources

Can’t get enough Collection info, check out these resources: