Re: LDAP sync users job

by Sam Moulem.  

Hello, 

I am testing the user sync and have the following issue:

Just one query in the user sync took more than 4 hours to complete.

I'd like to know if you have experienced the same thing.

I'm first testing with the cli before enabling the scheduled task in the web interface.

The cli script hanged; so I created a permanent 'tmp_extuser' table (as opposed to temporary table) so that our DBA can experiment with it.

I have edited auth/ldap/cli/sync_users.php 

I have commented out:     $ldapauth->sync_users(true);

And replaced with:             $ldapauth->sync_users(false);

I have edited the file auth/ldap/auth.php

In the function 'sync_users ';

I commented out:    $dbman->create_temp_table($table);

and replaced with:  $dbman->create_table($table);

I have also added a couple of debug output to see what steps were executes in the function  'sync_users ' before it hangs.

The 'User removal' steps are skipped; there are no users to remove.

The 'Revive suspended users' steps are skipped.

The 'User Updates' steps are skipped. 

We then get to the 'User Additions'

A SQL statement is prepared and the cli tool tries to execute it (but we never see the end):

"

$sql = 'SELECT e.id, e.username

                  FROM {tmp_extuser} e

                  LEFT JOIN {user} u ON (e.username = u.username AND e.mnethostid = u.mnethostid)

                 WHERE u.id IS NULL';

$add_users = $DB->get_records_sql($sql);

"

The output is as follow (ignore lines starting with 'sammy'):

"

..............Got 176729 records from LDAP

sammy - check user removal

sammy - no users to remove ==

sammy - check users to revive

sammy - no users to revive ==

sammy - # check 1

No updates to be done

sammy - # check 2

No updates to be done

sammy - start user addition

sammy - start building JOIN query

"

I connected to our MySQLdatabase and typed 'SHOW PROCESSLIST;'

The outcome is in the attached screenshot. But basically is says:

"

'mdluser_dev', 'localhost', 'moodle_dev_20170215', 'Query', '554', 'Sending data', 'SELECT e.id, e.username\n                  FROM mdl_tmp_extuser e\n                  LEFT JOIN mdl_use'

"

This is when I asked a DBA to look at it a created a permanent table. 

The DBA tried the query and it took more than 4 hours to complete:

"47215 rows in set (4 hours 23 min 58.46 sec)"

Our {user} table has '252580' rows

The  {tmp_extuser} has '176666' rows

I understand LDAP paged results were used (see comments in 'function sync_users($do_updates=true)' - file auth/ldap/auth.php). 

Which makes me think the tool will try to repeat the process more than once.

Initially we were thinking of enabling 'LDAP users sync job' which is in the 'Scheduled tasks' (on the web interface).

Now we think again

Any advice is welcome.

Thanks!