Setting up a FreeBSD svnsync mirror

There are several ways to setup a svn mirror. We could checkout everything with svn but our first checkout would take a lot of time (10+ hours with a fast connection!). In my opinion, setting up a mirror with seed files are the easiest and fastest way. 

First, I assume that we’ve installed subversion from packages or ports. I use /svnmirror as my svn source directory. 

If you use ZFS it would be a good idea to enable compression for this directory before use it (you can use other compression algorithms, gzip-9 mean best compression ratio but more cpu):

# zfs create −o compression=gzip−9 zfspool/svnmirror

We can run svn as root but I suggest a svn user, especially if we run a svnserve service later. You can do all steps as root but make sure, that svn is the owner and group of /svnmirror and all files in this directory at the end. If not, any update via cron will fail. 

pw groupadd svn
pw adduser svn -g svn -s /usr/sbin/nologin -d /svnmirror

mkdir -p /svnmirror (you can choose whatever directory you want)
cd /svnmirror
fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-base-r238500.tar.xz
fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-doc-r39237.tar.xz
fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/subversion/svnmirror-ports-r301235.tar.xz

Once we have downloaded all three files, we extract it. Make sure to have at least 20 GB of disk space before you extract all files. svnmirror-base-r238500.tar.xz will automatically create its subdirectory /base, svnmirror-doc-r39237.tar.xz its subdirectory /doc and so on. 

tar xvf svnmirror-base-r238500.tar.xz
tar xvf svnmirror-doc-r39237.tar.xz
tar xvf svnmirror-ports-r301235.tar.xz

/svnmirror# ls -1F

(we can delete all .xz files if we want. We don’t need them anymore)

Now, we update our tree with “svnsync sync file:///path/to/your/extracted/base“ that it fetches changes since the last revision in the archive:
svnsync sync file:///svnmirror/base
svnsync sync file:///svnmirror/doc
svnsync sync file:///svnmirror/ports

This process takes 30-60 minutes or more. It’s time for a coffee break ;)

Now we’ve updated copies of the repositories and we can setup a cron job to run our small script every 30 minutes to pull in new changes. 

% crontab −u svn −l

#minute hour mday month wday command
0/30 * * * * /usr/local/bin/svnsync sync -q file:///svnmirror/base
0/30 * * * * /usr/local/bin/svnsync sync -q file:///svnmirror/doc
0/30 * * * * /usr/local/bin/svnsync sync -q file:///svnmirror/port

The seed mirror is set to fetch from svn://svn.freebsd.org/base. The configuration for the mirror is stored in revprop 0 on our local mirror. To see the configuration:

svn proplist -v —revprop -r 0 file:///svnmirror/base

Use propset to change the svn mirror. As an example, we set it to svn0.us-west.freebsd.org:
svn propset svn:sync-from-url —revprop -r 0 svn://svn0.us-west.freebsd.org/base file:///svnmirror/base
% svn propset svn:sync-from-url —revprop -r 0 svn://svn0.us-west.freebsd.org/ports file:///svnmirror/doc
% svn propset svn:sync-from-url —revprop -r 0 svn://svn0.us-west.freebsd.org/ports file:///svnmirror/ports

We should mimick the UUIDs of the main FreeBSD Subversion repositories so that our users don’t have to check out a new working copy to accommodate the change: 

1. Get the official UUIDs :
% svn info svn://svn.FreeBSD.org/base | awk ‘/Repository UUID/{print $3}’
% svn info svn://svn.FreeBSD.org/doc | awk ‘/Repository UUID/{print $3}’
% svn info svn://svn.FreeBSD.org/ports | awk ‘/Repository UUID/{print $3}’

2. Set UUIDs:
% svnadmin setuuid /svnmirror/base ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
% svnadmin setuuid /svnmirror/doc c2e8774f-c49f-e111-b436-862b2bbc8956
% svnadmin setuuid /svnmirror/ports 35697150-7ecd-e111-bb59-0022644237b5

It should be the default setting but to be sure, check/svnmirror/{base,doc,port}/conf/svnserve.conf for anon-access = read. This line has to be set. At this point, we have a local mirror. Now it’s time to serve our repository to other machines. This can be done via ssh, http or svnserve. 

Serving with svnserve (easiest way)

Edit /etc/rc.conf and add:

svnserve_flags=”-d -R —listen-host”

The “−R” in the flags tells it to be read only. If you don’t want to listen to all IPs on your machine, then you can bind it so a specific one with replacing “—listen-host” with your IP. Default port is 3690. It can be changed with “—listen-port=PORT”.

Start svnserve via rc script and we are done.  Let’s try it: Checkout the latest source code for FreeBSD 9.1 with all security updates:

On the client: svn co svn://“IP_or_FQDN_from_our_svnserve”/base/releng/9.1 /usr/src

Let’s say we used svn.freebsd.org in the past for updating our FreeBSD 9.1 source code. We change the svn settings on all our clients to use our own svn mirror:
For FreeBSD 9.1
% cd /usr/src
% svn relocate svn://“IP_or_FQDN_from_our_svnserve”/base/releng/9.1

(it’s not necessary if you never checked out any source code with svn. In this case you can use: svn co svn://“IP_or_FQDN_from_our_svnserve”/base/releng/9.1)

For Ports:
% cd /usr/ports
% svn relocate svn://“IP_or_FQDN_from_our_svnserve”/ports/head

Cool, svn update connects to our own svn mirror.

Read www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/subversion-primer.html for more informaton.


CVS/CVSUP is deprecated in FreeBSD

CVS/CVSup is deprecated for FreeBSD 10+. For FreeBSD 9-stable, 8-stable and 7-stable, the FreeBSD Team will be attempting to continue updates through the exporter the official “support” end-of-life for last release on the branch. 

It’s time to start the migration from cvs to svn. In all my examples for the FreeBSD source code, I use /usr/src as root directory for svn and FreeBSD 9. You can use other root and FreeBSD versions.

Install subversion package:
If the ports tree is not available, Subversion can be installed as a package (old):
pkg_add −r subversion

If pkgng is being used to manage packages, Subversion can be installed with it instead (new):
pkg install devel/subversion

Or build it via ports source tree:
cd /usr/ports/devel/subversion; make install clean

Checkout FreeBSD source code:
svn checkout svn://svn.FreeBSD.org/repository/branch root

repository: base, ports, or doc.
branch: head, releng/9.1 release/9.1, stable/9
root: /usr/src, /usr/ports, /usr/doc

Checkout 9.1 including security patches
svn co svn://svn.freebsd.org/base/releng/9.1 /usr/src

Checkout to 9.1 release without any security patch since release (not recommended)
svn co svn://svn.freebsd.org/base/release/9.1 /usr/src

Checkout 9 Stable:
svn co svn://svn.freebsd.org/base/stable/9 /usr/src

Checkout Head (upcoming FreeBSD 10)
svn co svn://svn.freebsd.org/base/head /usr/src

After the first checkout we can a simple command to update our source tree:
cd /usr/src; svn update

Maybe you have already used svn for managing your FreeBSD 8.3. Now you want to upgrade to FreeBSD 9.1 via svn:
svn switch svn://svn.freebsd.org/base/releng/9.1/ /usr/src
cd /usr/src; svn update

Managing multiple source codes with svn
svn switch can also be used for managing several FreeBSD source trees at your system. You can download sources in example to /usr/src/stable/8, /usr/src/stable/9 (set root dir)… and switch between them with svn switch for managing later. 
When compiling the world, /usr/obj/ includes the name of the source tree, so if you have /usr/src/stable/9 and so on, you can use a single /usr/obj. You’ll get paths like /usr/obj/usr/stable/9 and so on. Handy if you want to play. 

Port update:
I recommend to use portsnap for updating your ports source tree.
portsnap fetch
portsnap extract <— only the first time
portsnap update

Checkout the latest Ports source tree via svn:
svn co svn://svn.freebsd.org/ports/head /usr/ports

Update Ports source tree via svn:
cd /usr/ports; svn update

For more information visit the official FreeBSD SVN Site and CvsIsDeprecated