No subject
Mon Jan 28 08:41:14 PST 2013
> Before each change to the OpenStack projects is merged into the main tree,
> unit and integration tests are run on the change, and only if they pass, is
> the change merged. We call this "gating".
There is a lot of debate about the value of a gated trunk on the internet;
which I'm not going to repeat here. OpenStack's experience has been that it
preserves the following properties;
http://openstack-ci.github.com/publications/lca2013-ci/index.html#(9)
- Ensures Code Quality
- Protects developers
- Devs always start from working code
- Protects tree
- Bad code doesn't land
- Egalitarian
- Process is the same for everyone
- Process is transparent
- Process is automated
These are all things that came up in Eric's "WebKit wishes" email specially
the parts about having an always green tree. The egalitarian nature of the
system also helps with trusting people as you *know* they can not break the
tree. This system is similar to our commit queue, however nobody
has privileges to bypass the queue.
OpenStack has 18 projects which are all tightly integrated, for example a
change in the API in one project could break another project, for this
reason they gate changes on tests runs from *all* projects before allowing
a commit to land to any of them. While WebKit is only a single project, the
process of requiring multiple jobs to be green is similar to WebKit needing
to support multiple platforms.
They do point out that when this system is set up, the system has to be
ultra repeatable and reliable;
> Once everything is automated, the projects stops if the automation does -
> http://openstack-ci.github.com/publications/lca2013-ci/index.html#(8)
To allow this to happen, OpenStack has managed to eliminated all flaky
tests in their suite. WebKit is not at this stage and still has a large
number tests which are both failing and/or flaky. Luckily, WebKit has much
better infrastructure for dealing with and tracking them down.
Other things they have done to try and make this process work are;
- Like WebKit, every patch is required to have code review before being
submitted. OpenStack requires two positive reviews before allowing a commit
to be submitted, rather than the single one that WebKit needs.
- Like WebKit, OpenStack has an "early warning system" which runs all
tests as soon as a patch is submitted.
The complete OpenStack test suite takes around ~1 hour to run, but as they
have more than 1 event per hour their landing system needs pipelining. They
have developed a system called Zuul to make this happen. Before they had
this pipeline process, committing was taking many hours to land.
You can see their currently running system at
http://zuul.openstack.org/ and find
out more about Zuul at the following locations;
> Zuul: a Pipelining Trunk Gating System
> http://amo-probos.org/post/14
http://mirror.linux.org.au/linux.conf.au/2013/ogv/OpenStack_Zuul.ogv
I guess this is something we should discuss further.
Tim 'mithro' Ansell
--f46d0434bf7205bef804d4eff1d3
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Hey guys,<div><br></div><div>Last week a number of the team here at Google =
Sydney, including myself attended <a href=3D"http://Linux.conf.au" target=
=3D"_blank">Linux.conf.au</a> 2013 conference. The conference was a blast a=
nd the hot topic this year was OpenStack, an Open Source Cloud layer.=C2=A0=
</div>
<div><br></div><div>The OpenStack project has grown from being a small proj=
ect to having over 500 active committers and continues to grow at a rapid p=
ace. Both the=C2=A0Continuous=C2=A0Integration Miniconf (<a href=3D"http://=
lca2013.linux.org.au/schedule/30102/view_talk?day=3Dmonday">http://lca2013.=
linux.org.au/schedule/30102/view_talk?day=3Dmonday</a>)=C2=A0and main confe=
rence included talks from OpenStack leaders about how they have tried to ha=
ndle this growth and I think we can learn from their successes and failures=
. All of the OpenStack's infrastructure is documented in the following =
talks=C2=A0<a href=3D"http://openstack-ci.github.com/publications/">http://=
openstack-ci.github.com/publications/</a>=C2=A0</div>
<div><br></div><div><div>I pulled the following stats to see how comparable=
the projects are;</div><blockquote style=3D"margin:0px 0px 0px 40px;border=
:none;padding:0px">OpenStack; (<a href=3D"http://openstack-ci.github.com/pu=
blications/lca2013-ci/index.html#(3)">http://openstack-ci.github.com/public=
ations/lca2013-ci/index.html#(3)</a>)</blockquote>
<div><ul><ul><li>Over 500 Active Technical Contributors</li><li>As many as =
200 trunk changes an hour</li><li>18 (integrated) projects (and growing)</l=
i></ul></ul></div><blockquote style=3D"margin:0px 0px 0px 40px;border:none;=
padding:0px">
I tried looking these up in WebKit and got the following;</blockquote><div>=
<ul><ul><li>~200 active contributors</li><li>As many as ~12=C2=A0trunk chan=
ges=C2=A0an hour</li><li>1 project, but 7 target platforms</li></ul></ul></=
div></div>
<div><div>One of the most interesting parts of OpenStack was having a "=
;gated trunk". From their talk;</div></div><blockquote class=3D"gmail_=
quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-=
color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Before each change to the OpenStack projects is merged into the main tree, =
unit and integration tests are run on the change, and only if they pass, is=
the change merged. =C2=A0We call this "gating". </blockquote><di=
v>
<div><br></div><div>There is a lot of debate about the value of a gated tru=
nk on the internet; which I'm not going to repeat here. OpenStack's=
=C2=A0experience has been that it preserves the following properties;=C2=A0=
<a href=3D"http://openstack-ci.github.com/publications/lca2013-ci/index.htm=
l#(9)">http://openstack-ci.github.com/publications/lca2013-ci/index.html#(9=
)</a></div>
<div><div><ul><li>Ensures Code Quality</li><li>Protects developers</li><ul>=
<li>Devs always start from working code</li></ul><li>Protects tree</li><ul>=
<li>Bad code doesn't land</li></ul><li>Egalitarian</li><ul><li>Process =
is the same for everyone</li>
<li>Process is transparent</li><li>Process is automated</li></ul></ul></div=
></div><div>These are all things that came up in Eric's "WebKit wi=
shes" email specially the parts about having an always green tree. The=
egalitarian nature of the system also helps with trusting people as you *k=
now* they can not break the tree. This system is similar to our commit queu=
e, however nobody has=C2=A0privileges=C2=A0to bypass the queue.</div>
<div><br></div><div>OpenStack has 18 projects which are all tightly integra=
ted, for example a change in the API in one project could break another pro=
ject, for this reason they gate changes on tests runs from <b>all</b> proje=
cts before allowing a commit to land to any of them. While WebKit is only a=
single project, the process of requiring multiple jobs to be green is simi=
lar to WebKit needing to support multiple platforms.</div>
<div><br></div><div><div>They do point out that when this system is set up,=
the system has to be ultra repeatable and reliable;</div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;bo=
rder-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Once everything is automated, the projects stops if the automation does -=
=C2=A0<a href=3D"http://openstack-ci.github.com/publications/lca2013-ci/ind=
ex.html#(8)">http://openstack-ci.github.com/publications/lca2013-ci/index.h=
tml#(8)</a></blockquote>
</div><div><br></div><div>To allow this to happen, OpenStack has managed to=
eliminated all flaky tests in their suite. WebKit is not at this stage and=
still has a large number tests which are both failing and/or flaky. Luckil=
y, WebKit has much better infrastructure for dealing with and tracking them=
down.=C2=A0</div>
<div><br></div></div><div>Other things they have done to try and make this =
process work are;</div><div><ul><li>Like WebKit, every patch is required to=
have code review before being submitted. OpenStack requires two positive r=
eviews before allowing a commit to be submitted, rather=C2=A0than=C2=A0the =
single one that WebKit needs.</li>
<li>Like WebKit, OpenStack has an "early warning system" which ru=
ns all tests as soon as a patch is submitted.</li></ul></div><div>The compl=
ete OpenStack test suite takes around ~1 hour to run, but as they have more=
=C2=A0than=C2=A01 event per hour their landing system needs pipelining. The=
y have developed a system called Zuul to make this happen. Before they had =
this pipeline process, committing was taking many hours to land.=C2=A0</div=
>
<div><br></div><div>You can see their currently running system at=C2=A0<a h=
ref=3D"http://zuul.openstack.org/">http://zuul.openstack.org/</a>=C2=A0and=
=C2=A0find out more about Zuul at the following locations;</div><blockquote=
class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:=
1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left=
:1ex">
Zuul: a Pipelining Trunk Gating System<br><a href=3D"http://amo-probos.org/=
post/14" target=3D"_blank">http://amo-probos.org/post/14</a></blockquote><b=
lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le=
ft-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;pad=
ding-left:1ex">
<a href=3D"http://mirror.linux.org.au/linux.conf.au/2013/ogv/OpenStack_Zuul=
.ogv">http://mirror.linux.org.au/linux.conf.au/2013/ogv/OpenStack_Zuul.ogv<=
/a>=C2=A0</blockquote><div>=C2=A0</div><div>I guess this is something we sh=
ould discuss further.</div>
<div><br></div><div>Tim 'mithro' Ansell</div><div><br></div><div><b=
r></div>
--f46d0434bf7205bef804d4eff1d3--
More information about the webkit-dev
mailing list