We had a 1s race condition where a user could get their rights loaded
at the exact time an admin is changing roles/caps and see the "old"
data. Or even see a half-updated view of the access controls.
Yuck.
So we fix the race condition backdating the dirtyness. Cheap, but
effective. And then we backdate it some more to cover for minor clock
flutter on clusters (you still need ntp however!).
accesslib: has_capability() now supports fake $USER for forum cron
has_capability() can handle the fake user that forum cron sets up
and will load the appropriate accessdata into $USER->access.
This makes forum cron work again. A test comparison between before
this patchseries yields:
With 1 forum post, sent total 24 times
- Before 11 000 DB queries (approx)
- After 506 DB queries
With 6 forum posts, sent a total of 452 times
- Before 47 876 DB queries
- After 8 256 DB queries
There is a very high variability, but we are going from 100-500
queries per sent email to 18-21 queries per sent email. The
variability probably stems from 2 of the 6 posts being in a 200-user
forum.
Still huge - by the time we are sending the email, we should know
everything we need to know about the user, the forum/thread/post and
the form. The average should be well below 1 DB query per email sent!
print_user_picture() was forcing an unneeded dbquery
if you need an imagealt. And who doesn't need one these days.
So - teach print_user_picture() to take either $userid
_or_ $userobj as the first parameter. If that first
parameter has the fields we need, never touch the db.
In other words, only touch the DB as a last resort.
There is a bit of ugliness in testing whether we have
the fields or not, because these fields are inconsistently
with/without NOT NULL in the DB definitions. So we cannot
use isset() because it barfs on nulls. And we cannot use empty()
because it will match both on "missing key" and ''.
And while at it, silence warnings that we are missing string
for the year(s). Also fixes a missing string bug in really boring
courses that noone's visitied in many years ;-)
Reworked course_parent_visible() to always return in a constant
number of db queries (2 worst case) regardless of nesting depth.
The rewritten version has a small cache, but if you are going to
walk many courses, it's still 1~2 DB queries per category seen,
so the right thing to do is to check it in the caller, as seen
in get_my_courses().
Reworked gmc to perform the course visibility checks. These are
very cheap if $CFG->allowvisiblecoursesinhiddencategories is true.
However, where we have to enforce category visibility, it adds a bit
of work. In simple terms, it adds a DB query to read all the categories,
and extra checks to make sure we are doing the right thing WRT
- course visibility vs the permission to see hidden courses
- category visibility vs the permission to see hidden categories
accesslib: has_cap_fad() - merge switchrole with defaultuserrole
If you are a teacher in course X, you have at least
teacher-in-X + defaultloggedinuser-sitewide. So in the
course you'll have techer+defaultloggedinuser.
We try to mimic that in switchrole.
Thanks to Petr for pointing me to a similar fix in CVS.
accesslib: drop rdef mangling for default role - check in has_cap_fad()
There are some exceptions when checking for caps that are inherited
from the default role. Move the check into has_cap_fad() and stop
mangling the data we put in $ad[rdef].
We now also set $ad[dr] to record default roles added.
This will later allow us to share rdef across many users in $ACCESS.
accesslib:load_all_capabilities() - fix guest user setup for multi-enrol
When setting things up for the guest user, the RA entry in accessdata
was not multi-enrol-friendly. Must have glossed it over in the
multi-enrol rework.
move_courses() now calls content_moved() which will take care of any
accesslib-required changes. And introducing move_category() which
also calls content_moved() when needed.
All interactive enrol/unenrol codepaths mark the context dirty
Manually enrolling and unenrolling self, and other users should
transparently set the context dirty. So walk all callers to
role_assign() and role_unassign() and mark the context dirty
where appropriate.
OTOH, most automated-backend enrol/unenrol mechanisms should not.
The backend lookups that happen when you login are well covered
by the login/enrolment process, and don't need to be marked dirty.
accesslib: get_user_courses_bycap() fix bug introduced by refactor
The refactor that created make_context_subobj() triggered a bug.
Smack in the hand to the sloppy programmer using variables outside
of the context they were meant to be used in!
admin/roles: context-specific role changes mark the context as dirty
And will force a reload of $USER->access for logged-in users that need
to read _this_ context. Much lower impact - still not a great idea to
edit assignments/caps on very busy courses, but impact should be low.
accesslib: Introduce functions to deal with dirty contexts
The accessinfo held in $USER->access can easily get out of
sync with reality if and admin has removed our access,
or expanded it after we loaded our accessinfo.
To handle this, we'll use the config_plugins table with an
'accesslib/dirtycontexts' plugin signature to store the paths of
recently changed contexts. To handle those dirrrty entries, here
we introduce
get_dirty_contexts() - for lib/setup
mark_context_dirty()
cleanup_dirty_contexts() - for cron
categories: Categories page and get_courses_page() - smarter about context
This patch saves 1600 context lookups on a 1600 course category. rcache
does help a bit with small categories, but on large setups, this is
not sustainable.
And it's not needed either. We have the data right at our fingertips.
Just get it when it's cheap...
accesslib: get_context_users_bycap() draft for course participants...
Introducing get_context_users_bycap() which gets the data in
2 DB queries, takes around 10ms on my laptop, and returns
the records with a nice context property attached.
Note Note Note: right now, some user recs do not have a context
associated, so are _not_ returned. So this awaits Matt's fix
to contexts maintenance to be 100% accurate...
First stage of role_switch() rewrite - role_switch() sets up
a rsw entry in $USER->access, and makes sure we have the
appropriate role definitions in rdef.
That is where get_role_access_bycontext() comes into play -
gets all the rdef entries in one cheap sql query.
... though there may be many of them...
TODO:
- fix callers of role_switch()
- teach has_cap_fromsess() to read the rsw entries
get_my_courses() and get_user_courses_bycap() field handling and caching
- Field handling moves back to get_my_courses() and now we have
almost all the fields that the old get_my_courses() did
(except for summary, which is *huge*) so get_my_courses() asks
for a lot of fields, but the get_user_courses_bycap() defaults
are _much_ leaner now.
I think this makes sense ;-)
- get_my_courses() now caches the course ids for the currently logged in
user in $USER->mycourses -- as a _string_. This is magnitudes more efficient
than having it as an array.
The cache makes a difference, but it's not very visible on
normal pageloads (with my courses block, for example).
However, over 100 iterations, for a user with 50 enrolments in a site
with 6K courses, we go from 4.3s to 0.6s. And the DB queries are *cheap*.
Using a combination of in-DB enrolments and in-session capability
checks, we narrow down the courses we fetch from the DB for checking.
This patch adds a small DB query, and has has a moderate impact on
the timings observable on my laptop (~300ms?), but reduces
*significantly* the bandwidth used, which in cluster environments
with frontends separate from backends has a serious impact.
accesslib: get_my_courses() rework, new get_courses_bycap_fromsess()
get_my_courses() goes from a bazillion queries (500 in some sample
cases) to 1 for the logged-in user, and 4 for a non-logged-in user.
One of those queries brings a *lot* of data across (all rows from
mdl_course) so there is room for serious optimisation.
However, this clocks at ~300 ms on my laptop, costly, but not
the end of the world. If your PHP-DB link has bandwidth probs
it might be a problem.
A few important changes to get_my_courses()
- (Compat ALERT!) the default fields are less than before --
(will be followed by patches that fix the callers!) our defaults
had grown to quite a bit because of the crazy caching scheme it had
- the $fields parameter is to name _additional_ fields you need, and
ideally wants them passed as an array. Will cope with old-style
strings too.
- the returned courses have an extra property "context" that is
a fully valid context object, so if the caller needs to perform
further accesslib checks, we save a query per course down the road
The work is done in the newfangled get_courses_bycap_fromsess()
which is brute-force but fast. I'm sure we can optimise the common
cases a lot in it if we try. It'd be worthwhile to
- reduce the rows we grab - that's really boneheaded
- if we copy and tweak the logic of has_cap_fromsess() in it
it can be made even faster
datalib:add_to_log() - avoid schema lookups, pace updates to user and user_lastaccess
* All updates to user.lastaccess and user_lastaccess.timeaccess are paced to
60s of the last update on the same record -- this should reduce the heat
on those tables.
* Updates/inserts to user_lastaccess are down with raw SQL to avoid costly
schema lookups on every request.
pagelib: Course page - preload child contexts and smarter user_allowed_editing()
With this patch we preload the child contexts for the course
and hold on to them. This means that in one DB query we have all the contexts
we are going to need.
The checks for user_allowed_editing() move from weblib:update_icon() to
user_allowed_editing(), where we cache the result, and in the process save
50% of the cap checks by testing separately blocks from modules (doh!).
Still, the cap checks here are very inefficient...
With the last 3 patches, a course page with default blocks and 9 modinstances
goes from 157 to 86 db queries when logged in as a non-editing user (guest,
student). As admin it drops from 88 to 81.
projects / moodle.git / log