hard code built in ignore attribute defs and names

From: Shilen Patel shilen@duke.edu
Sent: Friday, February 16, 2018 2:06 PM
To: Hyzer, Chris <mchyzer@isc.upenn.edu>
Cc: grouper-core@internet2.edu
Subject: Re: [grouper-core] RE: [grouper-users] StaleObjectStateException thrown by loader jobs after patch

Yeah I can’t imagine anybody would want to remove those builtins.

• Shilen

From: <grouper-core-request@internet2.edu> on behalf of "Hyzer, Chris" <mchyzer@isc.upenn.edu>
Date: Friday, February 16, 2018 at 1:37 PM
Cc: "grouper-core@internet2.edu" <grouper-core@internet2.edu>
Subject: [grouper-core] RE: [grouper-users] StaleObjectStateException thrown by loader jobs after patch

Core-team

Im kinda thinking that the builtins should be hard coded or configured somewhere else.

i.e. if someone overrides, they cant easily upgrade. Know what I mean? I think its ok for Grouper just to mandate we aren’t change logging certain things…

Thoughts?

Thanks
Chris

From: Shilen Patel shilen@duke.edu
Sent: Friday, February 16, 2018 1:33 PM
To: Scott Koranda <skoranda@gmail.com>; Hyzer, Chris <mchyzer@isc.upenn.edu>
Cc: grouper-users <grouper-users@internet2.edu>
Subject: Re: [grouper-users] StaleObjectStateException thrown by loader jobs after patch

Just to make sure, you do have the latest grouper.base.properties with this config:

grouper.attribute.namesOfAttributeDefNamesToIgnoreAuditsChangeLogPit.elConfig = $$grouper.attribute.rootStem$$:attestation:attestationCalculatedDaysLeft,$$grouper.attribute.rootStem$$:attestation:attestationLastEmailedDate,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastIncrementalMillisSince1970,$$grouper.attribute.rootStem$$:loaderMetadata:grouperLoaderMetadataLastSummary

(making sure it includes all the loaderMetadata attributes)

And you haven’t overwritten this config in grouper.properties. Just making sure.

Thanks!

• Shilen

On 2/16/18, 1:16 PM, "grouper-users-request@internet2.edu on behalf of Scott Koranda" <grouper-users-request@internet2.edu on behalf of skoranda@gmail.com> wrote:

> How many groups are loaded? Millions?

This morning I see this:

SQL> SELECT COUNT FROM GROUPER_GROUPS;
62221

SQL> SELECT COUNT FROM GROUPER_CHANGE_LOG_ENTRY_TEMP;
26791591

SQL> SELECT COUNT FROM GROUPER_CHANGE_LOG_ENTRY_TEMP WHERE STRING04 NOT LIKE 'etc:attribute:loaderMetadata:%';
0

SQL> SELECT COUNT FROM GROUPER_CHANGE_LOG_ENTRY;
6856091

There were no entries in either the change log or the change log temp
before patching.

I do not understand how so many entries were put into
GROUPER_CHANGE_LOG_ENTRY_TEMP by the new functionality.

I also drastically underestimated that amount of time it will take for
this deployment to process 27 million rows in the change log. It is
clear it will not be able to do it in a reasonable time (less than a few
weeks) so I am going to TRUNCATE the tables now instead.

> > Maybe I am not, however, understanding how those 27 million rows in
> > GROUPER_CHANGE_LOG_TEMP come about. Are they only created when the
> > "real" loader jobs run?
>
> Hmmmm. I deleted two of the new attributes, and change the incorrect value in a list of groups with two groups. I ran the loader. I got two change log entries per loaded group. One for etc:attribute:loaderMetadata:grouperLoaderMetadataLastSummary and one for etc:attribute:loaderMetadata:grouperLoaderMetadataLastFullMillisSince1970
> I believe you already have assignments on loaded groups for grouperLoaderMetadataLoaded and grouperLoaderMetadataGroupId.
>
> Not saying this is better, but before the patch, you could create those attributes, and run a GSH script to assign them to loaded groups, maybe a sleeping a second after a dozen assignments. Up to you.

I will consider it as an option. Thanks.

Another issue with these patches: every time a loader job runs it
generates thousands of log entries at the WARN level with a message like

2018-02-16 13:04:55,826: [DefaultQuartzScheduler_Worker-9] WARN
GrouperLoaderType.updateLoaderMetadataForGroupsNoLongerInLoader(2081) -

• Non-fatal error removing metadata on group:
  my:group:name:here

This is filling up the Grouper logs.

Is this expected?

Thanks,

Scott K