Beta and Candidate Releases
These instructions use outdated workflow based on svnmerge tool.
This flow takes the available changesets in trunk and brings checkpoint up to date. It assumes that trunk and checkpoint already exist and have been set up correctly.
The trunk is the authoritative source for everything that goes in a release; only under exceptional circumstances are changes made to other branches. The general rule is that a change is made to trunk and brought over to checkpoint.
This flow assumes that any needed changesets are already in trunk; if not, you will need to commit a suitable changeset following your usual development practices.
If you want a checklist you can mark off as you go, cut-n-paste these contents.
- Assumed Setup
- Send Starting Message
- Check Out Trunk
- Check Out Checkpoint
- Determine Available Changesets
- Edit Configuration Files
- Confirm Users' Guide is OK
- Commit Trunk
- Merge Trunk into Checkpoint
- Resolve Conflicts
- Update Release Files
- Commit Checkpoint
- Build Candidate Packages
- Check the Build
- Run Test Script
- Download SCons Home
- Prepare Blurb
- Prepare Tigris Announcement
- Prepare scons.org
- Tag the Release
- Archive Candidate Packages
- Update SourceForge
- Update scons.org
- Update the scons.org Web Site
- Update Tigris.org
These initializations are the conventions that will be used in the rest of this flow.
The location of the SCons SVN archive:
$ export HG=https://bitbucket.org/scons/scons
The version string, which should look something like this:
$ export RELEASE=3.2.0.beta.yyyymmdd
Send Starting Message
FIXME Send blurb out to release mailing list
Check Out Trunk
From within your base directory, execute this command:
$ svn co $SVN/trunk
Check Out Checkpoint
From within your base directory, execute this command:
$ svn co $SVN/checkpoint
Determine Available Changesets
FIXME Use svnmerge avail --log instead of dry run. Use svnmerge to find out what changesets are available to be merged:
$ cd checkpoint $ svnmerge avail --log >../trunk/log.file $ cd ../trunk
Edit Configuration Files
The commit.txt file created in the previous step has the commit comments for all the applied changesets. In theory, all of them have been included in the trunk files src/CHANGES.txt, stc/RELEASE.txt, and src/Announce.txt, but in practice, some of the information has not been included, so you will have to add it. At the same time, you want to prepare a log.file that will be used as the commit comments.
The log.file and src/CHANGES.txt files
Both log.file and src/CHANGES.txt use similar formats. Assuming commit.txt has three log entries, three from John Doe on two issues and two by Jim Smith on a single issue, log.file would look like this:
Rebased to trunk revision 54321: From John Doe: - One item that was done by John Doe. - Another item that was done by John Doe. From Jim Smith: - Item done by Jim Smith.
In theory, the first section of src/CHANGES.TXT should already have the corresponding information. If it does, steal it for log.file. If it doesn't, add it to src/CHANGES.txt. In the end, the items in src/CHANGES.txt for John Doe and Jim Smith should have this information:
From John Doe: - One item that was done by John Doe with details and examples. - Another item that was done by John Doe with details and examples. From Jim Smith: - Item done by Jim Smith with details and examples.
Verify the src/RELEASE.txt file
In theory, the entries in src/RELEASE.txt should already summarize src/CHANGES.txt except that issues are summarized by topic and the names of contributors are collected in a section at the bottom. In practice, not all of the entries have been transcribed.
Since this is an announcement, try to edit down the src/CHANGES.txt descriptions so that, ideally, each item fits on a single line. In general, get rid of examples and explanations. The idea is to pique the reader's interest, not provide full documentation. Still, use your judgment and feel free to leave in an explanation if it would be too confusing to shorten it.
Try to reorganize the items so that, within each section, all of the Visual Studio items are listed together, all of the Java items are together, etc. More "important" (or visible) items should appear towards the top of each section.
It's all right just to delete the description of something that's too difficult to explain and too minor to make much of a difference.
Whether or not a given item belongs in "NEW FUNCTIONALITY" vs. "CHANGED/ENHANCED" vs. "FIXED" vs. "IMPROVEMENT" is a judgment call.
In the final "Thanks to" paragraph, at a minimum list everyone in the src/CHANGES.txt file who contributed to the release, plus anyone else who (in your judgment) merits a mention even if they didn't contribute an actual patch.
The src/Announce.txt file
The src/Announce.txt file is the announcement used for production releases. In theory, it summarizes src/CHANGES.txt, but like src/CHANGES.txt, entries may not have been transcribed. Rather than trying to do the summarization all at once in a candidate release, it's better to do it incrementally while the changes to src/CHANGES.txt and src/RELEASE.txt are fresh in your mind.
Under the topic "scheduled for next release" add anything that's expected to be a significant change. To get a feel for what's in play, consult the tracker for issues scheduled for the next next release. The quickest way of getting that list is to go to the BugParty wiki page and click on the "how we're doing" link. On that report, click on the summary link at the bottom of the column for the next release. Use the future tense for these items.
Under the next topic, add any major items that aren't already included. Use your judgment as to what to add, but if it's not something the user needs to know, don't include it, so things like performance improvements, code cleanup, typos, and minor enhancements don't make the cut.
FIXME expand this section
Confirm Users' Guide is OK
CAUTION: Only try this step if you have the toolchain for building the documentation. The packages needed are discussed in DeveloperGuide/Documentation. It's quite a list.
The following steps verify that the user guide sources (some of which are generated from other files) are up to date.
FIXME (This will probably be reworked to be part of the build step.)
$ python bin/scons-doc.py --diff $ python bin/scons-doc.py --update
This step is needed only if any trunk files were modified. If nothing was changed, skip to the next step step.
From within the trunk directory:
Verify that version_tuple in ReleaseConfig has the correct release number and a release level of 'alpha'.
$ svn commit -F log.file Sending some/file Sending some/other/file Transmitting file data .. Committed revision 54321. $
Merge Trunk into Checkpoint
To merge the changes from the trunk into the checkpoint branch, run this command from within the checkpoint directory:
$ svnmerge merge -b -S $SVN/trunk -f commit.txt
FIXME This segment is overkill for anyone who actually uses SVN. Cut it or drop it.
A conflict occurs when SVN tries to apply changes from the archive that overlap with other changes made locally. These are rare but can happen when cherry-picking applies changes out of order. In general, the correct approach is to use the text from the archive rather than the local version, but use your judgment when making the changes as there will be some cases where the original text (or even a merger of the two texts) is the correct choice as SVN is not always accurate when marking the conflicted regions.
To determine if there're conflicts, run this command:
$ svn status ? test/example.py.r4941 ? test/example.py.mine ? test/example.py.r4649 C test/example.py ? src/engine/SCons/CoreFile.py.r4941 ? src/engine/SCons/CoreFile.py.mine ? src/engine/SCons/CoreFile.py.r4649 C src/engine/SCons/CoreFile.py $
Lines that begin with "C" identify a file that's conflicted. There are also a number of related files provided by SVN that can be used in the process of resolution, identified by lines that begin with "?". This is a simple resolution, so we won't use them.
To remove the conflicts from a file, follow these steps:
Locate a conflict marker in the text. The center of each conflicted region is marked with "=======" so searching for a line with only that on it will get you to the right place.
The start of the region is marked with "<<<<<<< .mine" and contains the local text. Delete this text as well as the start and center markers.
The end of the region is marked with something that looks like ">>>>>>> .r54321" and contains the text from the SVN archive. Delete the end marker.
- Repeat until there are no more conflict markers.
To see how your updated file compares with the archive, run this command:
$ svn diff test/example.py
If the changes aren't satisfactory, repeat the editing step.
Once the editing is done, use this command to tell SVN that the conflict has been resolved:
$ svn resolved test/example.py
As a side-effect, resolving the conflict will remove the related files provided by SVN.
Repeat this section for each conflicted file.
Update Release Files
Verify that the version_tuple in ReleaseConfig says 'beta' or 'candidate' as desired.
From within the checkpoint directory:
$ python bin/update-release-info.py release
The ReleaseConfig file is where the "official" version number ($VERSION), the Python version floors, and other information about the release is recorded. This command takes the information in ReleaseConfig and inserts it in the necessary files.
$ svn commit -m"Update checkpoint branch for $VERSION; see CHANGES.txt for details."
Build Candidate Packages
Provided you have all of the necessary utilities installed, this should be a simple matter of:
$ rm -rf build bootstrap $ python bootstrap.py
While you're waiting for the candidate packages to be built, open another shell window and proceed with preparing the test script. Make sure that $SVN and $VERSION are set up as shell variables as described above.
Prepare test script
The script to run the tests is in the source in bin/fill_me_in. Make a copy and edit it according to the instructions below.
# report skipped tests as successful export TESTCOMMON_PASS_SKIPS=yes set -e # stop on error # python runtest.py -a python runtest.py -a -p tar-gz python runtest.py -a -p zip python runtest.py -a -p local-tar-gz python runtest.py -a -p local-zip python runtest.py -a -p src-tar-gz python runtest.py -a -p src-zip python runtest.py -a -p rpm
The regression suite exercises a wide range of functionality. There are around a thousand tests in the suite, so each test run can take a while.
Regression tests of source tree
Normally, the SCons BuildBot monitors this branch, so it probably started running regression tests as soon as you checked in the changes above. If your machine is covered by one of the BuildBot machines (particularly if your machine is one of the BuildBot machines), you can remove the first runtest in the script.
Regression tests of candidate packages
The build step above not only builds the packages but also unpacks all of them into subdirectories so that you can run the test suite against the packaged goods. This is intended to catch packaging problems such as not adding a new module to the packaging MANIFEST list(s). The runtest.py script supports a -p option and arguments that run the SCons tests against the different unpacked directories.
Edit the script to include the tests you want to include. If you want to be complete, test all of the packages.
To be quicker but still reasonably thorough, test tar-gz and zip, one each of local- and src- (probably do -tar-gz for one and -zip for the other), and rpm.
For a quick-n-dirty test, just test tar-gz or zip, and maybe rpm. Since all of the different packages use the same lists as input, it's pretty unlikely that the tests will pass on one package and fail for another.
Once the script is prepared, save it and return to the window creating the packages.
Check the Build
The build creates the packages in the build/dist subdirectory. If everything built correctly, you should see files that look like the following:
scons-$VERSION-1.noarch.rpm scons-$VERSION-1.src.rpm scons-$VERSION.linux-x86_64.tar.gz scons-$VERSION.linux-x86_64.zip scons-$VERSION.tar.gz scons-$VERSION.win32.exe scons-$VERSION.zip scons-doc-$VERSION.tar.gz scons-local-$VERSION.tar.gz scons-local-$VERSION.zip scons-src-$VERSION.tar.gz scons-src-$VERSION.zip
Note that the linux-x86_64 strings in some of the file names above may be different, depending on your local system architecture. That doesn't matter; those files don't go public. Go ahead and remove them.
Run Test Script
If the build succeeded, run the test script:
$ sh path/to/copy-of-script
The tests can run a very long time. While you're waiting, go back to your other shell window and prepare what you can on the assumption that the tests will succeed.
Download SCons Home
From within your base directory, run these commands:
$ svn co $SVN/scons.org $ cd scons.org
The scons.org checkout is pretty large. If you don't want to wait, open another shell window and proceed. Make sure the new shell is set up with the SVN and VERSION shell variables as described above.
Prepare the blurb by starting with a copy of build/scons/RELEASE.txt. It should be pretty much good-to-go, but look over it and see if there are any changes that should be made to make it more suitable as a blurb rather than release notes in a distribution. In general, use your own judgment as to what to keep and what to cut.
FIXME HTML copy of blurb by inserting <br/><br/><br/> instead of double blank lines, <br/><br/> instead of single blank lines, and <br/> in front of items in a list. Not perfect, but more readable.
FIXME Source of beta and final blurbs are different.
Prepare Tigris Announcement
Note that these files must be checked in to the trunk for them to show up on our tigris.org project pages; you can not update and commit these files from the directory in which you built the packages.
From within the www subdirectory of your trunk directory, edit these files:
Add a short highlight announcement that will appear at the top of all of the scons.tigris.org pages. Trim the list to remove any entries that are now obsolete.
Update the text at the top to reflect that this release is now the latest available release.
If you opened another window to do the prep above, move back to the window downloading scons.org; it should be done by now. If you didn't, change back to the scons.org subdirectory of your base directory.
Make the following changes to the following files: FIXME The destination names aren't CHANGES.txt or RELEASE.txt and the explanation needs to be tweaked. And the source for RELEASE.txt could be either RELEASE.txt or Announce.txt.
copy the new file from build/scons/CHANGES.txt in your packaging directory
copy the new file from build/scons/RELEASE.txt in your packaging directory
update $latestrelease to the value $VERSION FIXME
add an announcement for the home page
add an announcement to the list (duplicate it from what you just added to index.php)
To unpack the release documentation into where it can be used by the web pages, just run this script:
FIXME When this script is checked in, it will allow arguments on the command line
$ sh bin/FillMeIn [ $VERSION [ $SVN ] ]
or something similar. In the meantime, copy the script and run it.
mkdir doc/$VERSION (cd doc/$VERSION && tar zxf build/dist/scons-doc-$RELEASE.tar.gz) svn add doc/$VERSION (cd doc && rm -f latest && ln -s $VERSION latest) case $VERSION in *.0.final.*) # This is a 'final' release on the main branch (cd doc && rm -f production && ln -s $VERSION production) esac
Tag the Release
From within the checkpoint directory:
$ svn cp . $SVN/tags/$VERSION
Archive Candidate Packages
Verify that you have SVN and VERSION in your shell environment as described in the setup section above, then run the bin/scp-sourceforge script:
$ sh bin/scp-sourceforge sf_username
Mark packages with release notes
Navigate to the SourceForge File Manager page for SCons. If you're reading this from a printed copy rather than the wiki page and can't click on the link, follow these instructions:
Log in to your SourceForge account
Click on Develop just under Forge in the SourceForge banner.
Under My Projects, click on Develop next to the SCons project.
Click on Project Admin -> File Manager
Open the scons, scons-src, and scons-local directories. Within each one, open the $VERSION folder. In the scons/$VERSION directory, select the RELEASE.txt file and mark mark it as the release notes in the popup that appears, then save.
If this is a final release of the production branch, select scons-$VERSION.win32.exe and mark it as the default selection for Windows, then select scons-$VERSION.tar.gz and mark it as the default for all other OSes.
In turn, select each of the files (other than the release notes file itself, which is automatically set) in all three folders (and the folders themselves) and specify that RELEASE.txt is to be the release note for that file. Because of the way the page refreshes after clicking Save, it seems to be easier to start at the bottom and work your way up.
FIXME: TEST THIS: You can do multiple releases quickly by opening the File Manager page multiple times in a separate tabs, but if you do, make sure to wait for the page to reload completely in one tab before clicking Save in another. Trying to update multiple releases at once doesn't work, presumably because the session can only handle one update at a time.
Hide previous releases
- For a checkpoint release, hide all checkpoints in the same series that are older than this checkpoint.
- For a minor or micro release, hide all the checkpoint releases leading up to this release.
- For a major release, hide all the checkpoint releases leading up this release AND hide all major and minor releases older than this release.
For each release name you wish to hide, you have to do the following in scons, scons-local, and scons-src:
Click on the gear icon to the left of the package name and select Cut from the popup.
Click on the gear icon to the left of old checkpoints and select Paste from the popup.
Navigate to the SCons download page. If you're reading this from a printed copy rather than the wiki page and can't click on the link, follow these instructions:
FIXME Add instructions if can't click.
For each of the scons, scons-src, and scons-local directories:
- Open the directory.
- Open the $VERSION folder within the directory.
- For each package in the folder, click on the link to download it.
- Verify that the files arrived and look reasonable.
Add news item
Navigate to the Project News page. If you're reading this from a printed copy rather than the wiki page and can't click on the link, follow these instructions:
FIXME These aren't complete.
Go to the Project Admin -> Feature Settings page
In Project News, click Submit.
Once you get to the Project News page:
Fill in the Subject: box with something like "Release $VERSION now available"
Cut-and-paste the blurb you prepared above into the Details: box
Update the scons.org Web Site
Apply changes to web site
Commit the changes you prepared above:
$ svn commit -m"Changes to publish the SCons $VERSION release"
Now you have to go update the site:
$ ssh -l scons manam.pair.com $ cd public_html $ cp -al production new # hard-link the whole tree; fast. $ (cd new && svn up) # should be fast here too $ rm -rf previous $ mv production previous && mv new production $ exit
Now point your browser to the SCons home page. If anything is amiss, fix it, commit the necessary changes, and update the site.
Navigate to the SCons download page.
- Verify that the opening paragraph makes sense.
- In the right-hand sidebar, find the download section corresponding to $VERSION.
- For each file in the section, click on the link to download it.
- Verify that the files arrived and look reasonable.
Update project pages
Send the changes you prepared above to Tigris:
From within the www subdirectory of your trunk directory:
$ svn commit -m"Update project web pages for $VERSION"
Point your browser at the roadmap page; if anything's amiss, fix it and commit again.
Add news item
Log in to your tigris.org account
Click Announcements in the left-hand nav bar
Click Add new announcement
- Double-check the date (it's probably already set)
Fill in the Headline box
Fill in the Body box with the HTML blurb
Click Add new announcement
Add release name to issue tracker
Click Issue Tracker on the left-hand nav bar
Click Configuration options
Click Add/edit components
Under scons, to the far right of Add ..., click Version
At the bottom of the list, to the right of "Add a new version", click Add
Fill in the Version: box with the release ($VERSION)
Click the Add button