[ATrpms-devel] yum-presto

[Jonathan Dieter]

On Tue, 2007-05-08 at 22:02 +0200, Axel Thimm wrote:
> Hi Jonathan,
> 
> thanks for the extensive info, see some questions inline.
> 
> > First off, I want to state that this system is still very much in beta,
> > but I'm using it to create presto repositories for fc6 updates, extras
> > and livna (i386, x86_64), devel (i386) and CentOS 5 updates (i386).
> 
> Does that mean that the presto repos can be on a different server than
> the full rpms?
Yes.  All that's needed is a mirror of the atrpms repository.

> 
> > As far as I can tell, it won't be used by FI until F8 at the
> > earliest, so I just get to keep on building them on the test server.
> 
> Well, that's the part that makes me a bit suspicious - while at first
> sight it looks like a non-intrusive extension, having Fedora ban it
> for F8 or later indicates that some FI folks consider this more pain
> than gain ATM.
Fair enough, though I think the word "ban" is a bit harsh.  The problem,
at least as far as I can see it, is that most of the people in Fedora
who would need to push and test this are really busy right now with the
merge and will be ready to focus on this after F7 is out.  I honestly
expect that FI will start generating at least some drpms for Rawhide at
some point in the pre-F8 cycle (though I could be wrong).

> 
> > createprestorepo uses approximately 3x MB of RAM (where x is the size of
> > the package in MB) and will use quite a bit of CPU power (especially for
> > the larger packages like openoffice.org-core).  It does need at least
> > two rpms to create a deltarpm, so you would have to work out some
> > workaround to get around that problem with your repository scripts.
> 
> Can those rpms be in completely different folders like say
> 
> /production/online/atrpms-stable
> 
> /staged/preparing-next-update/atrpms-stable
> 
> My scripts try to be as atomically as possible, so they prepare a new
> repo (using cached metadata for creatrepo) and swap it in place.
> 
Yes, but you'd have to do something like:
cd /production/all_drpms/
ln -s /production/online/atrpms-stable old
ln -s /staged/preparing-next-update/atrpms-stable new
makedeltarepo ./ DRPMS/    # Create deltarpms
rm old
prunedrpms ./              # Remove any old drpms (see below)
rm new
createprestorepo ./        # Create presto metadata

Makedeltarepo is smart enough to only make deltarpms for the packages
that have changed.  Prunedrpms only removes drpms that upgrade *to* an
rpm that doesn't exist any more.

> > (I'd be thinking along the lines of soft links to both old and new
> > directories, but I think there might be some bugs to work out with that
> > idea.)
> > 
> > To answer the whole N*(N-1)/2 deltas question, the answer is no.
> > Rather, if you have N old packages (or N-1 old deltas pointing to an old
> > package + the old package), createprestorepo will create N new deltas
> > pointing at the new package.  prunedrpms will remove any deltas pointing
> > to rpms that are no longer in your repository.
> 
> Just to understand the concept: If there are N old packages and N-1
> deltas and I add a new package (w/o yet reoving an old one), what
> happens? Do I get N deltas from all old packages to the new one? Or
> only on delta from the newest old package to the new one and presto
> constructs the packages by merging together several deltas?
> 
> If the latter is true, e.g. "chained deltas" then prunedrpms removing
> intermediate deltas would not be good. :)
> 
Let's say you had foo-1.0, foo-1.1, and foo-1.2 in your repository
(through a normal series of upgrades).  At the moment, you only have
foo-1.2.rpm and two drpms, foo-1.0_1.2.drpm and foo-1.1_1.2.drpm.

When you put foo-1.3 in your repository, makedeltarepo will first
generate foo-1.2_1.3.drpm, and then combine foo-1.0_1.2.drpm and
foo-1.2_1.3.drpm to create foo-1.0_1.3.drpm.  It will then do the same
to create foo-1.1_1.3.drpm.

At this point you have two rpms (foo-1.2 and foo-1.3) in your repository
and five drpms (two that upgrade to foo-1.2 and three that upgrade to
foo-1.3).  If you remove foo-1.2.rpm and then run prunedrpms, it will
remove foo-1.0_1.2.drpm and foo-1.1_1.2.drpm.  It will leave
foo-1.0_1.3.drpm, foo-1.1_1.3.drpm and foo-1.2_1.3.drpm.  These are the
only drpms that will be needed when foo-1.4 comes out.

I hope all of that was clear.

> > Don't know if that answers your questions.  To give you an idea of
> > space, FC6 i386 Extras is taking up roughly 7 GB of space and only has
> > 200MB of deltarpms.  FC6 i386 Updates is taking up roughly 6 GB of
> > space, but has far more deltarpms (900MB) because of more rapid updates.
> 
> I don't think there are more updates in updates than in Extras, there
> are just less old packages in Extras that justify these numbers.
You're probably right.  I'm not sure what your numbers will look like,
but I'm guessing that we won't know until you give it a shot.

> 
> > Hope this gives you an idea of what the requirements for presto are.
> 
> Thanks!

You're most welcome.

Jonathan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.atrpms.net/pipermail/atrpms-devel/attachments/20070509/a15d5608/attachment-0001.bin