Cherry-Picking from Trunk to Checkpoint
These instructions use outdated workflow based on svnmerge tool.
Cherry-picking is the act of applying only selected changesets to a current branch, rather than taking all possible changesets to bring the branch up-to-date.
In our case, trunk is the authoritative source for everything that goes in a release; only under exceptional circumstances are changes made to checkpoint or trunk. The general rule is that a change is made to trunk and brought over to checkpoint.
This flow assumes that the needed changesets are already in trunk; if not, you will need to commit a suitable changeset to trunk following your usual development practices prior to using this flow.
- Assumed Setup
- Check Out Checkpoint
- Determine Revision(s)
- Merge Revision(s) from Trunk
- Resolve Conflicts
- Update Release Values
- Commit Modified Checkpoint
- Build Candidate Packages
- Check the Build
- Run Test Script
- Determine What and Where to Announce
- Download SCons Home
- Prepare Blurb
- Prepare Tigris Announcement
- Prepare scons.org
- Confirm Tests Succeeded
- Tag the Checkpoint
- Archive Candidate Packages
- Update SourceForge
- Update Scons.Org
- Update Tigris.Org
These initializations are the conventions that will be used in the rest of this flow.
The location of the SCons SVN archive:
$ export HG=https://bitbucket.org/scons/scons
The version string, which should look something like this:
Check Out Checkpoint
From within your base directory, execute this command:
$ svn co $SVN/checkpoint
You can review the changes available on trunk to be integrated into checkpoint:
$ svnmerge.py avail --log -S $SVN/trunk
You can look at the diff that was applied by this revision:
$ svnmerge.py avail --diff -S $SVN/trunk -r54321
Use this information to determine the revision(s) to be applied.
Merge Revision(s) from Trunk
After selecting the revision(s), merge them into the checkpoint branch.
From within the checkpoint directory:
$ svnmerge.py merge -r54321,54123,53412 -b -S $SVN/trunk -f commit.txt
FIXME This segment is overkill for anyone who actually uses SVN. Cut it or drop it.
A conflict occurs when SVN tries to apply changes from the archive that overlap with other changes made locally. These are rare but can happen when cherry-picking applies changes out of order. In general, the correct approach is to use the text from the archive rather than the local version, but use your judgment when making the changes as there will be some cases where the original text (or even a merger of the two texts) is the correct choice as SVN is not always accurate when marking the conflicted regions.
To determine if there're conflicts, run this command:
$ svn status ? test/example.py.r4941 ? test/example.py.mine ? test/example.py.r4649 C test/example.py ? src/engine/SCons/CoreFile.py.r4941 ? src/engine/SCons/CoreFile.py.mine ? src/engine/SCons/CoreFile.py.r4649 C src/engine/SCons/CoreFile.py $
Lines that begin with "C" identify a file that's conflicted. There are also a number of related files provided by SVN that can be used in the process of resolution, identified by lines that begin with "?". This is a simple resolution, so we won't use them.
To remove the conflicts from a file, follow these steps:
Locate a conflict marker in the text. The center of each conflicted region is marked with "=======" so searching for a line with only that on it will get you to the right place.
The start of the region is marked with "<<<<<<< .mine" and contains the local text. Delete this text as well as the start and center markers.
The end of the region is marked with something that looks like ">>>>>>> .r54321" and contains the text from the SVN archive. Delete the end marker.
- Repeat until there are no more conflict markers.
To see how your updated file compares with the archive, run this command:
$ svn diff test/example.py
If the changes aren't satisfactory, repeat the editing step.
Once the editing is done, use this command to tell SVN that the conflict has been resolved:
$ svn resolved test/example.py
As a side-effect, resolving the conflict will remove the related files provided by SVN.
Repeat this section for each conflicted file.
Update Release Values
Verify that release_tuple in ReleaseConfig has the correct release and says 'beta' or 'candidate' as appropriate, then run the update-release-info command:
$ python bin/update-release-info.py release
The ReleaseConfig file is where the "official" version number ($VERSION), the Python version floors, and other information about the release is recorded. This command takes the information in ReleaseConfig and inserts it in the necessary files.
Commit Modified Checkpoint
Prepare a log.file that describes the revisions you pulled over. You can use the commit.txt returned by the merge step to help prepare the log file. Don't just reuse commit.txt as it tends to be very verbose and hard to read.
Here's an example of a reasonable log.file:
Cherry-picked revisions from trunk because blah blah blah. From John Doe: - One item that was done by John Doe. - Another item that was done by John Doe. From Jim Smith: - Item done by Jim Smith.
Use your judgment. If all the revision(s) from John Doe were documentation, and Jim Smith fixed an important bug (and was the actual reason for the cherry-picking), the log message might be like this:
Jim Smith fixed the yadda yadda by blah blah, and the issue was deemed important enough to be brought over into the checkpoint branch on a expedited schedule toward a micro release. Also picked up documentation updates from John Doe as they have no possibility of destabilizing the release.
Then commit the result to SVN:
$ svn commit -F log.file Sending some/file Sending some/other/file Transmitting file data .. Committed revision 54321. $
Build Candidate Packages
Provided you have all of the necessary utilities installed, this should be a simple matter of:
$ rm -rf build bootstrap $ python bootstrap.py
While you're waiting for the candidate packages to be built, open another shell window and proceed with preparing the test script. Make sure that $SVN and $VERSION are set up as shell variables as described above.
Prepare test script
The script to run the tests is in the source in bin/fill_me_in. Make a copy and edit it according to the instructions below.
# report skipped tests as successful export TESTCOMMON_PASS_SKIPS=yes set -e # stop on error # python runtest.py -a python runtest.py -a -p tar-gz python runtest.py -a -p zip python runtest.py -a -p local-tar-gz python runtest.py -a -p local-zip python runtest.py -a -p src-tar-gz python runtest.py -a -p src-zip python runtest.py -a -p rpm
The regression suite exercises a wide range of functionality. There are around a thousand tests in the suite, so each test run can take a while.
Regression tests of source tree
Normally, the SCons BuildBot monitors this branch, so it probably started running regression tests as soon as you checked in the changes above. If your machine is covered by one of the BuildBot machines (particularly if your machine is one of the BuildBot machines), you can remove the first runtest in the script.
Regression tests of candidate packages
The build step above not only builds the packages but also unpacks all of them into subdirectories so that you can run the test suite against the packaged goods. This is intended to catch packaging problems such as not adding a new module to the packaging MANIFEST list(s). The runtest.py script supports a -p option and arguments that run the SCons tests against the different unpacked directories.
Edit the script to include the tests you want to include. If you want to be complete, test all of the packages.
To be quicker but still reasonably thorough, test tar-gz and zip, one each of local- and src- (probably do -tar-gz for one and -zip for the other), and rpm.
For a quick-n-dirty test, just test tar-gz or zip, and maybe rpm. Since all of the different packages use the same lists as input, it's pretty unlikely that the tests will pass on one package and fail for another.
Once the script is prepared, save it and return to the window creating the packages.
Check the Build
The build creates the packages in the build/dist subdirectory. If everything built correctly, you should see files that look like the following:
scons-$VERSION-1.noarch.rpm scons-$VERSION-1.src.rpm scons-$VERSION.linux-x86_64.tar.gz scons-$VERSION.linux-x86_64.zip scons-$VERSION.tar.gz scons-$VERSION.win32.exe scons-$VERSION.zip scons-doc-$VERSION.tar.gz scons-local-$VERSION.tar.gz scons-local-$VERSION.zip scons-src-$VERSION.tar.gz scons-src-$VERSION.zip
Note that the linux-x86_64 strings in some of the file names above may be different, depending on your local system architecture. That doesn't matter; those files don't go public. Go ahead and remove them.
Run Test Script
If the build succeeded, run the test script:
$ sh path/to/copy-of-script
Determine What and Where to Announce
The sequel describes all the steps for a full-fledged beta or candidate release. This could be overkill. For example, if the change is small and has no impact on the way SCons operates (such as a fix to a regression test so that a production release may proceed), there's no reason to publish the checkpoint at all, as publishing the final release will be sufficient.
There's usually a reason for the choice of changesets to be cherry-picked, and it's important to someone. At a minimum, those interested parties should be notified. Other forms of publishing are optional, so use your judgment as to which, if any, of the following steps should be done.
The tests can run a very long time. While you're waiting, go back to your other shell window and prepare what you can on the assumption that the tests will succeed.
Download SCons Home
From within your base directory, run these commands:
$ svn co $SVN/scons.org $ cd scons.org
The scons.org checkout is pretty large. If you don't want to wait, open another shell window and proceed. Make sure the new shell is set up with the SVN and VERSION shell variables as described above.
Prepare the blurb by starting with a copy of build/scons/RELEASE.txt. It should be pretty much good-to-go, but look over it and see if there are any changes that should be made to make it more suitable as a blurb rather than release notes in a distribution. In general, use your own judgment as to what to keep and what to cut.
FIXME HTML copy of blurb by inserting <br/><br/><br/> instead of double blank lines, <br/><br/> instead of single blank lines, and <br/> in front of items in a list. Not perfect, but more readable.
FIXME Source of beta and final blurbs are different.
Prepare Tigris Announcement
Note that these files must be checked in to the trunk for them to show up on our tigris.org project pages; you can not update and commit these files from the directory in which you built the packages.
From within the www subdirectory of your trunk directory, edit these files:
Add a short highlight announcement that will appear at the top of all of the scons.tigris.org pages. Trim the list to remove any entries that are now obsolete.
Update the text at the top to reflect that this release is now the latest available release.
If you opened another window to do the prep above, move back to the window downloading scons.org; it should be done by now. If you didn't, change back to the scons.org subdirectory of your base directory.
Make the following changes to the following files: FIXME The destination names aren't CHANGES.txt or RELEASE.txt and the explanation needs to be tweaked. And the source for RELEASE.txt could be either RELEASE.txt or Announce.txt.
copy the new file from build/scons/CHANGES.txt in your packaging directory
copy the new file from build/scons/RELEASE.txt in your packaging directory
update $latestrelease to the value $VERSION FIXME
add an announcement for the home page
add an announcement to the list (duplicate it from what you just added to index.php)
To unpack the release documentation into where it can be used by the web pages, just run this script:
FIXME When this script is checked in, it will allow arguments on the command line
$ sh bin/FillMeIn [ $VERSION [ $SVN ] ]
or something similar. In the meantime, copy the script and run it.
mkdir doc/$VERSION (cd doc/$VERSION && tar zxf build/dist/scons-doc-$RELEASE.tar.gz) svn add doc/$VERSION (cd doc && rm -f latest && ln -s $VERSION latest) case $VERSION in *.0.final.*) # This is a 'final' release on the main branch (cd doc && rm -f production && ln -s $VERSION production) esac
Confirm Tests Succeeded
If the BuildBot display and the test run(s) are OK, then proceed below to publishing the release. If not, fix the problem in trunk following your usual development practices and go back and start again.
Tag the Checkpoint
Verify that you have SVN and VERSION in your shell environment as described in the setup section above, then run this command:
$ svn cp . $SVN/tags/$VERSION -m"Tagging $VERSION"
Archive Candidate Packages
Verify that you have SVN and VERSION in your shell environment as described in the setup section above, then run the bin/FillMeIn script:
set -e # stop on error rm -rf ARCHIVE svn co --depth=files $SVN/ARCHIVE mkdir ARCHIVE/scons-$VERSION cp build/dist/* ARCHIVE/scons-$VERSION svn add ARCHIVE/scons-$VERSION svn commit -m"Saving copy of $VERSION in archive" ARCHIVE rm -rf ARCHIVE
Verify that you have SVN and VERSION in your shell environment as described in the setup section above, then run the bin/scp-sourceforge script:
$ sh bin/scp-sourceforge sf_username
Mark packages with release notes
Navigate to the SourceForge File Manager page for SCons. If you're reading this from a printed copy rather than the wiki page and can't click on the link, follow these instructions:
Log in to your SourceForge account
Click on Develop just under Forge in the SourceForge banner.
Under My Projects, click on Develop next to the SCons project.
Click on Project Admin -> File Manager
Open the scons, scons-src, and scons-local directories. Within each one, open the $VERSION folder. In the scons/$VERSION directory, select the RELEASE.txt file and mark mark it as the release notes in the popup that appears, then save.
If this is a final release of the production branch, select scons-$VERSION.win32.exe and mark it as the default selection for Windows, then select scons-$VERSION.tar.gz and mark it as the default for all other OSes.
In turn, select each of the files (other than the release notes file itself, which is automatically set) in all three folders (and the folders themselves) and specify that RELEASE.txt is to be the release note for that file. Because of the way the page refreshes after clicking Save, it seems to be easier to start at the bottom and work your way up.
FIXME: TEST THIS: You can do multiple releases quickly by opening the File Manager page multiple times in a separate tabs, but if you do, make sure to wait for the page to reload completely in one tab before clicking Save in another. Trying to update multiple releases at once doesn't work, presumably because the session can only handle one update at a time.
Hide previous releases
- For a checkpoint release, hide all checkpoints in the same series that are older than this checkpoint.
- For a minor or micro release, hide all the checkpoint releases leading up to this release.
- For a major release, hide all the checkpoint releases leading up this release AND hide all major and minor releases older than this release.
For each release name you wish to hide, you have to do the following in scons, scons-local, and scons-src:
Click on the gear icon to the left of the package name and select Cut from the popup.
Click on the gear icon to the left of old checkpoints and select Paste from the popup.
Navigate to the SCons download page. If you're reading this from a printed copy rather than the wiki page and can't click on the link, follow these instructions:
FIXME Add instructions if can't click.
For each of the scons, scons-src, and scons-local directories:
- Open the directory.
- Open the $VERSION folder within the directory.
- For each package in the folder, click on the link to download it.
- Verify that the files arrived and look reasonable.
Add news item
Navigate to the Project News page. If you're reading this from a printed copy rather than the wiki page and can't click on the link, follow these instructions:
FIXME These aren't complete.
Go to the Project Admin -> Feature Settings page
In Project News, click Submit.
Once you get to the Project News page:
Fill in the Subject: box with something like "Release $VERSION now available"
Cut-and-paste the blurb you prepared above into the Details: box