Re: [Xen-devel] [xen 4.6 retrospective] [bad] review load near freeze point

[Lars Kurth <lars.kurth.xen@xxxxxxxxx>]

Jan,
I wanted to pick this one up again, as this also came up in the developer 
meeting (where we did a face-2-face retrospective). A few other possible 
solutions were bounced regarding this problem.
Regards
Lars

> On 12 Aug 2015, at 09:00, Jan Beulich <jbeulich@xxxxxxxx> wrote:
> 
>>>> On 04.08.15 at 14:52, <lars.kurth.xen@xxxxxxxxx> wrote:
>   = Issue / Observation =
> 
> Maybe my memory regarding the 4.5 release has faded, but I'm
> having the impression that 4.6 was quite a bit worse. This was at
> parts due to the sheer number of patch series and their sizes,
> but also because frequently they took quite a bit more than just a
> couple of iterations.
> 
> Some of this is (as mentioned on past discussions) caused by too
> little review involvement of non-maintainers (the need of which
> exists because of limited bandwidth maintainers have for doing
> reviews), but I think there are also quality issues here (see below).

This also came in the retrospective at the developer meeting, on Wed the 19th. 
I believe that the following items contributed to this (I want to cover the 
"not enough review capacity" and "quality issues" separately).

A) Too many things happening around the freeze
B) Not enough coordination amongst committers
C) Poor communication: the community didn't read Wei's emails on the change of 
the release process and was surprised by it

>   = Possible Solution / Improvement =
> 
> While I agree that some aspects (like coding style issues) could be
> taken care of by requiring submitters to have their patches be
> checked by (not yet existing) tools/scripts, I think much of this is a
> problem of enforcing discipline on oneself: Sloppiness in style,
> according to my observation, frequently is accompanied by
> sloppiness in the actual coding. While I think that suggestion may
> be considered harsh, I'd like to throw up for discussion a cut off
> on the number of iterations that may be submitted for a particular
> series within a certain time (with a high threshold on exceeding
> the limit for really minor adjustments). I would hope that making
> people aware that the number of tries they have is limited, they
> would start self-reviewing their changes more consciously.
> 
> And yes (to preempt respective responses), I'm aware that such a
> rule may raise the bar for first time contributors. But those often
> don't contribute large patches/series, and I don't think such a rule
> would need to be enforced as strictly on small contributions (plus
> newcomers could be granted some [limited] slack).
> 
> And of course, people should submit their code early, and follow
> up diligently to review comments. Submitting a v1 half a year
> before the freeze, and being at say v4 around freeze time (and
> then perhaps even demanding the code to go in because "it was
> submitted early") is about as bad as starting work on a big series
> a month ahead of freeze.

A few other possible solutions, which may help alleviate the freeze point 
crunch may be:

== 1) Highlight any changes to the process in the release manager's boilerplate 
in every release mail ==

This would fix C)

== 2) Get rid of freeze extensions altogether ==

This should shorten the stressful period a little; albeit the effect may be 
that the stress may start somewhat earlier and help with A)
Even though in the short term, this may not immediately lead to contributors 
planning to get patches in at the beginning of the release cycle, in the long 
run, this should do it. 

== 3) Earlier hard freeze for large and risky patches ==

Another thing we could do, is have an earlier freeze point for large and risky 
patches. Aka, anything of a certain size or high risk has to go in earlier than 
the hard freeze, allowing for some re-work or tidy up between that early freeze 
point and the final one. It could would act as a filter and create some focus. 
As such it should help with A) and B). Whether we allow exceptions, between 
those two "freeze" points is an open question.

== 4) Better upfront planning of what is desired to be in a release. In 
particular for larger patches. ==

Part of the reason why 4.6 was so stressful, had to do with sheer numbers: Xen 
4.5 had 1812 commits, and 4.6 had 1974 commits after the push gate cleared. We 
had a number of large patch series (at least 50% of the ones which were close 
and caused some of the stress) where we started designs in the 4.5 release 
cycle, with contributors hoping they would go in.

That number of features was unusually high. If in Feb we had known that there 
way say 10 large/complex series coming towards the end of the cycle, and say 2 
were from vendor X, 4 from vendor Y, ... we could ask the community and each 
vendor, to sort these larger ones by priority. 

Not quite sure how the community prioritisation would work in practice. Asking 
vendors to prioritise their own planned patches is obviously easier.

We probably can get some data that tells us how many of these patch series we 
can on average into good shape in a release cycle. And build in some slack.

The release manager and reviewers could then track the subset more closely. I 
think related to this is that the fair/good/... tagging is maybe not that 
useful. It may be worth thinking about different ways of tagging these.

Playing devils advocate:
Let's just assume we had known upfront, which series were more "important" than 
others. Warning flags to contributors could have been raised earlier along the 
lines of: "You highlighted this patch as important, but the patch is only at v2 
now and should be much further. Unless, we there is significant progress in X 
weeks, we would take that series out of the 'important' bucket.". It would also 
create more focus earlier and encourage maintainers to review those "important" 
patches earlier. So this should help with A) and B).

Of course this may not work that well in practice. However, any software 
development team does that type of prioritisation and then tightens what they 
do closer to the release.    

Regards
Lars


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel