Bulk build notes
Joerg Sonnenberger's new bulk building interface is located in pkgtools/pbulk. On pkgbox, this is located in /archive/NetBSD-pkgsrc.
To do this without affecting the existing pkgsrc setup on the machine you are using, you create a chroot and install pkgsrc twice inside it - once for pbulk and the various support programs, and once for the actual pkgsrc where everything is built.
These notes are taken from Justin Sherrill's test builds on pkgbox.dragonflybsd.org, and from suggestions from Joerg.
Setting up the chroot
If getting rid of an earlier install, delete all the files you can, adjust the permissions on the ones left, and then delete what remains.
rm -rf /build/pbulk_chroot chflags -R noschg /build/pbulk_chroot rm -rf /build/pbulk_chroot
Create dir for chroot
mkdir /build/pbulk_chroot
Add files to chroot
cd /usr/src
If there was no prior buildworld in this , you'll need to do this step
make DESTDIR=/build/pbulk_chroot buildworld
Add world to chroot
make DESTDIR=/build/pbulk_chroot installworld
note that this is not /etc, but /usr/src/etc
cd etc make DESTDIR=/build/pbulk_chroot distribution
Additional files that will be needed in the chroot
cp /etc/resolv.conf /build/pbulk_chroot/etc
Make sure you've loaded the linux module if you don't want to get unnecessary messages about unknown ELF types.
kldload linux.ko
Jeremy C. Reed has a large repository of the distfiles for pkgsrc in another directory outside of the chroot; mounting it and adding it to DIST_PATH should make it available, saving the bulk build process (and Matt's bandwidth) from having to download more.
mkdir /build/pbulk_chroot/archive mount_null /archive /build/pbulk_chroot/archive
Should be ready to go...
chroot /build/pbulk_chroot /etc/rc.d/ldconfig start
Adding pkgsrc bootstrap
Everything past this point is in the chroot!
mkdir /bulklog mkdir /scratch mkdir /distfiles mkdir /packages
Grab the most recent quarterly release of pkgsrc:
setenv CVSROOT anoncvs@anoncvs.NetBSD.org:/cvsroot setenv CVS_RSH ssh cd /usr cvs -q checkout -rpkgsrc-2008Q1 -P pkgsrc
To update the pkgsrc files later
cvs -q update -dP
Put this in /root/mk-base.conf, for use later when building the first bootstrap:
WRKOBJDIR = /scratch PKGSRCDIR = /usr/pkgsrc DISTDIR = /distfiles DIST_PATH = /archive/distfiles PACKAGES = /packages FAILOVER_FETCH= yes _ACCEPTABLE= yes ALLOW_VULNERABLE_PACKAGES= yes PKG_DEVELOPER?= yes
We need to create a clean bootstrap kit for later. This installs pkgsrc tools into /usr/pkg_bulk, too.
mkdir /usr/pkg_bulk/ cd /usr/pkgsrc/bootstrap ./bootstrap --gzip-binary-kit bootstrap.tgz --mk-fragment=/root/mk-base.conf ./cleanup ./bootstrap --pkgdbdir /usr/pkg_bulk/.pkgdb --prefix /usr/pkg_bulk cd /usr/pkgsrc/pkgtools/pbulk /usr/pkg_bulk/bin/bmake install
We need to make a binary kit with the corrected mk.conf so that pbulk is always set up with the right version as it builds/removes/adds packages
cd /usr/pkgsrc/bootstrap mkdir temp mv /bootstrap.tgz temp cd temp tar xf bootstrap.tgz cp /usr/pkg/etc/mk.conf usr/pkg/etc/mk.conf tar hzcf ../bootstrap.tar.gz usr var
Edit /usr/pkg_bulk/etc/pbulk.conf and change these lines to what is needed. Be sure to put in a destination for the rsync command, as a side effect of that process is the removal of packages that are not for distribution due to various laws. If this is being performed on a single machine, nfsmount directories in the chroot so that you can still copy the files around. The rsync options below assume the packages and report are being copied around on the local machine.
base_url=http://something/you_should_change_here master_mode=no pkg_rsync_args="-av --delete-excluded" pkg_rsync_target="/archive/packages/DragonFly-1.12/pkgsrc-2008q1" report_rsync_args="-avz --delete-excluded" report_rsync_target="/archive/packages/DragonFly-1.12/pbulk_report" report_recipient="you@your.email" make=/usr/pkg/bin/bmake # optional keep_wrkdir=no keep_prefix=no
Kick off the bulk build. It takes a LONG time, so it's worth doing this in a screen session. Unless you install screen in the same way as pbulk, it won't be available in the chroot, so you may need to get out of the chroot, enter a screen session, and re-enter the chroot before this step.
/usr/pkg_bulk/bin/bulkbuild
The line for setting up a rsync server for the mirrors: (do this outside the chroot)
rsync --daemon --bwlimit=200 --config=/usr/pkg/etc/rsync/rsyncd.conf
If the build stops or pauses or whatever, this line will restart it, for versions 0.28 and older.
/usr/pkg_bulk/libexec/pbulk/build
Version 0.29 and later restart the build with this:
/usr/pkg_bulk/bin/bulkbuild-restart