Performance concerns with Loader Jobs using Privileges settings

As an example, I have a loader job with 840K members over 65K groups (courses at UF). Many of the courses of Reader, Viewer priv with one group (the same group for each). The priv group is within the 65K groups. If I don't use Privs and just manage the membership, the loader job takes about 35 min. With the Privs (groups metadata) as part of the loader job, it takes 2 hours. 90 min of processing time is spent just managing the groups metadata. This processing added/deleted no members. Would love to see processing time made the same - 35 min with or without Privs.