qlnx-psets 0.5

qlnx-psets is a software born to help dealing with the nice data generated by the combination of bkcvs repository and cvsps.

  Add it to your Download Basket!

 Add it to your Watch List!


Rate it!
send us
an update
GPL (GNU General Public License) 
3.3/5 14
Davide Libenzi
ROOT \ Utilities
qlnx-psets is a software born to help dealing with the nice data generated by the combination of bkcvs repository and cvsps. The data that is dumped on the CVS repository, while being a complete mirror of the real bk repository, it is quite difficult to work out due to the limitations of CVS in the operations that I am usually doing on such data. Those typically leads to cheacking out e given version (on a patch set base) and adding/removing patch sets.

The qlnx-psets tool uses cvsps to collect a log and a series of patch sets, and it uses such data to generate user configurations. It might be worth to be noticed, that the qlnx-psets tool can be used on any CVS repository, not only Linux kernels (with some limitations regarding kernel-version -> patchset mapping).

Setup and Use

To start using qlnx-psets, you need to start by installing cvsps.

Download the package and install it by following the instructions given by the authors (and say thanks to them since cvsps really brings back order to the CVS caos). The whole process starts by rsync'ing the bkcvs CVS repository from kernel.org, that in my machines usually goes like:

$ cat bksync.sh

rsync -vzrlt --delete rsync.kernel.org::pub/scm/linux/kernel/bkcvs /usr/src/

The shell script above fetches the bkcvs CVS repository into the /usr/src/bkcvs directory. Since rsync is based upon a smart protocol, updates of the local repository are typically fast and not bandwidth hungry. The qlnx-psets tool supports a few command line options, that can also be fed using a configuration file (to avoid retyping):

qlnx-psets.pl [--qroot DIR] [--root DIR] [--repo NAME] [--outdir DIR]
[--cfgfile FILE] [--no-cvs-direct] [--no-bkcvs] [--hard-links] [--help]
cmd [arg [, arg]]

--qroot DIR = Sets the tool root directory to DIR (mandatory)
--root DIR = Sets the CVS root to DIR (mandatory)
--repo NAME = Selects the repository inside the CVS root to NAME (mandatory)
--outdir DIR = Sets the output directory to DIR (default .)
--cfgfile FILE = Use the config file FILE to load options
--no-cvs-direct = Unselect the CVS-direct mode of cvsps
--no-bkcvs = Unselect the special --bkcvs mode of cvsps
--hard-links = Use hard links to replicate caches
--help = Prints this help

Mapping between command line options and configurations file options are:

--qroot => $qlnx_qroot (QLNX_QROOT)
--root => $qlnx_root (QLNX_ROOT)
--repo => $qlnx_repo (QLNX_REPO)
--outdir => $qlnx_outdir (QLNX_OUTDIR)
--no-cvs-direct => $qlnx_cvsdirect (either empty or "--cvs-direct") (QLNX_CVSDIRECT)
--no-bkcvs => $qlnx_bkcvs (either empty or "--bkcvs") (QLNX_BKCVS)
--hard-links => $qlnx_usehl (QLNX_USEHL)

My example configuration file looks like:

# Maps to --qroot (defaults to environment QLNX_QROOT)
$qlnx_qroot = "/home/nobkup/qlnx-psets";

# Maps to --root (defaults to environment QLNX_ROOT)
$qlnx_root = "/usr/src/bkcvs";

# Maps to --repo (defaults to environment QLNX_REPO)
$qlnx_repo = "linux-2.5";

# Maps to --outdir (defaults to environment QLNX_OUTDIR)
$qlnx_outdir = ".";

# Maps to --no-cvs-direct (defaults to environment QLNX_CVSDIRECT)
$qlnx_cvsdirect = "--cvs-direct";

# Maps to --no-bkcvs (defaults to environment QLNX_BKCVS)
$qlnx_bkcvs = "--bkcvs";

# Maps to --hard-links (defaults to environment QLNX_USEHL)
$qlnx_usehl = 1;

The qlnx-psets.pl root directory (--qroot) is the place where the tool generate its own metadata to help and speedup the user command processing. Inside such directory, the following structure can be found:

caches < dir >
patches < dir >
xpatches < dir >

The "caches" directory stores cached copies of the tree, to help cutting down the generation time of user configurations. Inside the "caches" directory, one or more directory will be stored by the user, with name equal to the patchset number they cache precedeed by the '#', or the version string they refer to. For example, my current "caches" contains only one cached copy "#18121", that corresponds to 2.6.6 (will see later how):

[davide@bigblue qlnx-psets]$ ll caches/
total 4
drwxrwxr-x 18 davide davide 4096 May 9 19:33 #18121

Inside cache directories we will find the typical content of the cached tree:

[davide@bigblue qlnx-psets]$ ll caches/#18121/
total 284
drwxrwxr-x 22 davide davide 4096 May 9 19:31 arch
-rw-r--r-- 1 davide davide 18691 May 9 19:32 COPYING
-rw-r--r-- 1 davide davide 85686 May 9 19:32 CREDITS
drwxrwxr-x 2 davide davide 4096 May 9 19:33 crypto
drwxrwxr-x 42 davide davide 4096 May 9 19:33 Documentation
drwxrwxr-x 44 davide davide 4096 May 9 19:33 drivers
drwxrwxr-x 53 davide davide 4096 May 9 19:33 fs
drwxrwxr-x 33 davide davide 4096 May 9 19:32 include
drwxrwxr-x 2 davide davide 4096 May 9 19:33 init
drwxrwxr-x 2 davide davide 4096 May 9 19:33 ipc
drwxrwxr-x 3 davide davide 4096 May 9 19:33 kernel
drwxrwxr-x 4 davide davide 4096 May 9 19:33 lib
-rw-r--r-- 1 davide davide 51590 May 9 19:32 MAINTAINERS
-rw-r--r-- 1 davide davide 36156 May 9 19:32 Makefile
drwxrwxr-x 2 davide davide 4096 May 9 19:33 mm
drwxrwxr-x 32 davide davide 4096 May 9 19:33 net
-rw-r--r-- 1 davide davide 13976 May 9 19:33 README
-rw-r--r-- 1 davide davide 2815 May 9 19:32 REPORTING-BUGS
drwxrwxr-x 7 davide davide 4096 May 9 19:33 scripts
drwxrwxr-x 3 davide davide 4096 May 9 19:33 security
drwxrwxr-x 15 davide davide 4096 May 9 19:33 sound
drwxrwxr-x 2 davide davide 4096 May 9 19:33 usr

I could rename the directory caches/#18121 to caches/2.6.6 and the effect would result to be the same. When used together with the bkcvs-generated data, caches should be retrieved directly from the CVS root, and not from kernel tarballs. This because the kernel tarballs is missing of some directories/files that patchsets wants to patch. Failing in doing so will result in errors from the patching operations. To do this you need to checkout from the CVS root by the date of the release you want to checkout. To do this, supposing we want to checkout release 2.6.5, we first search the logs for such release:

$ qlnx-psets.pl sl 'Tag: v2_6_5'

This will dump all log entries that contains such tag, and at the end you will see something like:

PatchSet 17248
Date: 2004/04/04 04:34:07
Author: torvalds
Branch: HEAD
Tag: v2_6_5
Linux 2.6.5

BKrev: 406f822foQTgmLYKWBzPlaKSfZ_q_g


This means that such tag has been created on 2004/04/04. So we can then use such date to checkout from CVS:

$ cd caches
$ cvs -d /usr/src/bkcvs co -D 2004-04-04 linux-2.5
$ mv linux-2.5 2.6.0

Other caches can simply be created with the qlnx-psets tool itself, by creating custom configuration file like:

$ cat 2.6.6.conf

and by issuing:

$ qlnx-psets.pl cs 2.6.6.conf
$ mv linux-2.5 $QLNX_QROOT/caches/2.6.6

The qlnx-psets tool uses some heuristic to find out the best cache to use to generate a given user configuration. Care should be taken when using the --hard-links option because the generated copy uses hard link to the cached copy. The qlnx-psets tool automatically break the link when applies the patchsets, but the user should be careful when editing the built configuration. We will show later how to find mappings between patchset numbers and kernel revisions. The "patches" directory is the place where cvsps generated patch files wil be stored. File names inside the "patches" directory have the format:


where PATCHSETNUM is the patchset that has to be used to migrate to version PATCHSETNUM. The "xpatches" directory stores patches that do not come from the CVS patch set, but that are instead coded by the user, and that the user wants to apply to his configuration. The "lastps" file is used by the tool to store the last patchset synced from the CVS repository, in order to cut update times. Only new patchset files will be generated when ussuing a 'cupdate' command. The "logfile" is a cached copy of the full cvsps log file, and it is used to speed up operations that might be time bound to the cvsps processing performance. The "migfile" is generated by the qlnx-psets tool, by parsing the "logfile" and detecting associations between patchset numbers and kernel versions. An example few lines of my "migfile" are:

2.6.5 17248
2.6.6-rc1 17705
2.6.6-rc2 17841
2.6.6-rc3 17985
2.6.6 18121
2.6.7-rc1 18806
2.6.7-rc2 19020
2.6.7-rc3 19355

We see here that, for example, the patchset 18121 correspond to the kernel version 2.6.6 (the one that is shown cached in the lines above). The "psets" file stores, one by line, all the patchsets contained inside the cvsps log file. To start using qlnx-psets, you can just edit the supplied configuration file and edit the variables to fit your setup. Typically you might want to set $qlnx_qroot, $qlnx_root, $qlnx_repo and $qlnx_outdir. As first-time caching speed up trick, you can edit (create actually) the "lastps" file containing the oldest patchset that you will ever request to remove. For example, if you know you configuration will never go back past version 2.6.2, you can set the "lastps" to 15325 (patchset version corresponding to 2.6.2). Then you can start issuing your first request to qlnx-psets, and this will be the one that will first create and sync its metadata with the CVS repository selected inside the configuration file:

$ ./qlnx-psets.pl --cfgfile qlnx.config cu

The command will take some time, because the qlnx-psets tool will ask cvsps to generate all patchsets starting from the "lastps" one up to the most current one (note that if you didn't set the "lastps" file, this can really take some time). The good news is, that new updates will always take a fraction of the time the first one took (depending on how many new patchsets have been generated inside the selected CVS repository).

Last updated on April 8th, 2008


#qlnx psets #bkcvs repository #data generator #qlnx #psets #bkcvs #cvsps

Add your review!