[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Solving the gitlab-ci git fetch issue, was: [PATCH 3/3] automation: add a QEMU based x86_64 Dom0/DomU test


  • To: Stefano Stabellini <sstabellini@xxxxxxxxxx>
  • From: Anthony PERARD <anthony.perard@xxxxxxxxxx>
  • Date: Thu, 28 Oct 2021 10:48:20 +0100
  • Authentication-results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none
  • Cc: <cardoe@xxxxxxxxxx>, <xen-devel@xxxxxxxxxxxxxxxxxxxx>, <iwj@xxxxxxxxxxxxxx>, <wl@xxxxxxx>, <andrew.cooper3@xxxxxxxxxx>, "Stefano Stabellini" <stefano.stabellini@xxxxxxxxxx>
  • Delivery-date: Thu, 28 Oct 2021 09:48:52 +0000
  • Ironport-data: A9a23:fT3Ffqybw3MvIIp6Y0h6t+eqwSrEfRIJ4+MujC+fZmUNrF6WrkUPn WUeUDrTMvqIajfzKtwibd609h8FvZLWy9NhQAVvqyAxQypGp/SeCIXCJC8cHc8zwu4v7q5Dx 59DAjUVBJlsFhcwnvopW1TYhSEUOZugH9IQM8aZfHAuLeNYYH1500s6w7Rp2tcAbeWRWGthh /uj+6UzB3f9s9JEGjp8B3Wr8U4HUFza4Vv0j3RmDRx5lAa2e0o9VfrzEZqZPXrgKrS4K8bhL wr1IBNVyUuCl/slIovNfr8W6STmSJaKVeSFoiI+t6RPHnGuD8H9u0o2HKN0VKtZt9mGt8JBm OUKpYXhcEQSJ6nTlb4NcxJkDBgraMWq+JefSZS+mcmazkmAeHrw2fR+SkoxOOX0+M4uXzsIr 6ZBbmlQMFbT3Ipaw5riIgVors0lMMnsOpJZonx6xCvVJf0nXYrCU+PB4towMDIY25ASQ6mCO 5pxhTxHcgbpQi9qPgouUMgXo8ChqCDtfj1apwfAzUYwyzeKl1EguFT3C/LUZd6iVchThlyfp G/N4yL+GB5yHMOb4SqI9DSrnOCntTjnVYwbErr+7eJjilCXwm0VCTUGVF+6p+n/gUm7M/pEM FAd8Ccqqak09WSoQ8P7Uhn+p2SL1jYXUsBcFOAS4wiXxq3ZpQ2eAwAsXjNHLdArqsIybTgrz UOS2cPkAyR1t7+YQm7b8a2bxQ5eIgBMczVEP3VdC1JYvZ+z++nfky4jUP5xIP6Fyc3sSAiz+ CmOsyVkl5UYsdEygvDTEU/8vxqgoZ3ATwgQ7wrRX3644g4RWLNJd7BE+nCAsq4edNfxok2p+ SFewZDHvb9m4YSlzXTVGI0w8KeVC+Fp2dE2qWVkGIU97HyT8nqncJE4DNpWdRoxbJhsldMEZ ibuVeJtCH17YCTCgUxfOdvZ5yEWIU7ITo2Nuhf8NYMmX3SJXFXblByCnGbJt4wXrGAikLskJ bCQetu2AHARBMxPlWTtGrxDieVwmHFumQs/oKwXKTz8iNJyg1bOEN843KamNLhlvMtoXi2Mq 76zyPdmOz0ACbajM0E7AKYYLEwQLGhTOHwFg5c/SwJ3GSI/QDtJI6aImdsJItU594wIxrag1 izsASdwlQug7UAr3C3XMxiPnpu0Bs0hxZ/6VARxVWuVN48LOtfysv1HLcJrLdHKNoVLlJZJc hXMQO3Yatwnd9gN0211gUDVoNMweRK1qxiJOib5MjEzc4Q5H17C+8P+fxup/y4LV3Llucw7q rym9wXaXZtcGFgyUJeIMKqinwGroHwQuONuRE+UcNNdT1rhrdpxICvrg/5pf8xVcUffxiGX3 hq9CAsDobWfuJc89dTE3PjWr4qgH+ZkMFBdGm3XseS/OSXApzLxyo5cSueYOzvaUTqsqqmlY OxUydD6MeEGwwkW49YtTe4zwPtntdX1prJcwgB1J1nxbgymWuF6P32L/chTrakRlLVXjhS7B xCU8d5ANLTXZM68SAwNJBAoZ/io3O0PnmWA9uw8JUj36XMl/LeDVkkObRCAhDYEcelwOYIhh +wgpNQX+0q0jR9zaoSKiSVd9mKtKH0cUvp46sFGUdGz0gd7mEtfZZH8CzPt5MDdYtpBBUAmP zuIifeQnL9b3EfDLyI+GHWlMTCxXnjSVMSmFGM/Gmk=
  • Ironport-hdrordr: A9a23:+Yl2zKGA62x5gytGpLqE6MeALOsnbusQ8zAXP0AYc3Jom+ij5q STdZUgpHrJYVkqNU3I9ertBEDEewK6yXcX2/hyAV7BZmnbUQKTRekIh7cKgQeQeBEWntQts5 uIGJIeNDSfNzdHsfo=
  • Ironport-sdr: 8tFzC1kL3YfRIxKhIZoaS0VkzHMCMocbjjwMPSJSSBBv06NwH/OOQN+qPu03dTqlAiulAvvr/w z60GILbu56ZpNqLaM46b3+2Xw3LcNB8BjIfsNQEJ9kTImvki9RhQo0YYauqYbf7QdCHqNEr4yR yJ/Qu7UpzLTCCq4oa05Fh8+s5Fp5t3OJSjkLsPfPyc/Cbmxwq3SlaQuKk721fevrH4+iN8uOwC TN+aG5U4gNh7hE7QmmD247kCscoThL7LWhGB7R0O/YVLlCgvDQM0OebwFkS7m3UjGzVZrqAPYV WunIZRSaJ2eDjG79PEdaPtk4
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On Wed, Oct 27, 2021 at 02:43:46PM -0700, Stefano Stabellini wrote:
> On Wed, 27 Oct 2021, Anthony PERARD wrote:
> > > But we do have a severe problem at the moment with external sources: our
> > > "git clones" keep failing during the build on x86. That is definitely
> > > something worth improving (see my other email thread on the subject) and
> > > it is the main problem affecting gitlab-ci at the moment, I keep having
> > > to restart jobs almost daily to get the overall pipeline to "pass".
> > > 
> > > If you have any ideas on how to stop fetching things using "git" from
> > > external repositories in gitlab-ci that would be fantastic :-)
> > > The only thing I could think of to "fix it" is moving all external repos
> > > to gitlab repositories mirrors.
> > 
> > I don't think that would work, I've seen the initial clone/fetch of a
> > job fail as well, so from gitlab. If we could have a cache of those
> > external resources closer to the runners, that would be better.
> 
> You mean like a git repository mirror inside the Rackspace network (the
> provider of the x86 runner), right? Then we would force the git client
> to go to the Rackspace mirror instead of directly to the target using
> "insteadOf".

That would seems the best to me. If we could install Ian's
git-cache-proxy that is used in osstest, that would be good I think.
Having a mirror instead might work too but that would mean figure out
which repo we would need a mirror of.

I did try a different alternative a while back, I tried to use gitlab's
caching capability:
    automation: Cache sub-project git tree in build jobs
    
https://lore.kernel.org/xen-devel/20191219144217.305851-3-anthony.perard@xxxxxxxxxx/
It mostly works but I'm not sure how useful it is as it seems there is
10 computers that would maintain 10 different caches, and most of them
for a short while.

> Is that what you meant? Doug, do you think it would work?

-- 
Anthony PERARD



 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.