Details
-
Improvement
-
Resolution: Done
-
Minor
-
None
-
None
Description
Large number of changes are best handled by a FullSync, perhaps being 2-10x more efficient.
1) Create a provisioner-configuration setting incrementalSyncChangeThreshold, default 2500, which indicates that PSPNG should schedule a FullSync after it has seen 2500 changes for a single group through incremental provisioning. It is expected that some changes beyond this threshold would be handled incrementally while the FullSync request is queued, etc.
2) Keep in memory or in grouper attributes: the last full-sync time of each group
3) incremental processing should compare the dates of incremental events and ignore the ones that were created before the last full sync.
The end result will be that FullSync will occur in the middle of a large stream of incremental events, eventually be finished, and then the end of the stream of incremental events will immediately marked as completed.