Uploaded image for project: 'Grouper'
  1. Grouper
  2. GRP-2017

PSPNG - automatically process large change streams as full syncs

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Done
    • Minor
    • 2.4.0.patch
    • None
    • provisioning
    • None

    Description

      Large number of changes are best handled by a FullSync, perhaps being 2-10x more efficient.

       

      1) Create a provisioner-configuration setting incrementalSyncChangeThreshold, default 2500, which indicates that PSPNG should schedule a FullSync after it has seen 2500 changes for a single group through incremental provisioning. It is expected that some changes beyond this threshold would be handled incrementally while the FullSync request is queued, etc.

      2) Keep in memory or in grouper attributes: the last full-sync time of each group

      3) incremental processing should compare the dates of incremental events and ignore the ones that were created before the last full sync.

       

      The end result will be that FullSync will occur in the middle of a large stream of incremental events, eventually be finished, and then the end of the stream of incremental events will immediately marked as completed.

       

      Attachments

        Activity

          People

            bert.beelindgren@at.internet2.edu Bert Bee-Lindgren (gatech.edu)
            bert.beelindgren@at.internet2.edu Bert Bee-Lindgren (gatech.edu)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: