Sound advice - blog

Martin Pool has begun a venture into constructing a new version control system called bazaar-NG. At first glance I can't distinguish it from CVS, or the wide variety of CM tools that have come about recently. It has roughly the same set of commands, and similar concepts of how working files are updated and maintained.

This is not a criticism, in fact at first glance it looks like we could be seeing a nice refinement of the general concepts. Martin himself notes:

I don't know if there will end up being any truly novel ideas, but perhaps the combination and presentation will appeal.

To the end of hopefully contributing something useful to the mix, I thought I would describe the CM system I use at work. When we initially started using the product it was called Continuus Change Management (CCM), and has since been bought by Telelogic and rebadged as CM Synergy. Since our earliest use of the product it has been shipped with a capability called Distributed Change Management (DCM), which has since been rebadged Distributed CM Synergy.

Before I start, I should note that I have seen no CM synergy source code and have only user-level knowledge. On the other hand, my user level knowledge is pretty in-depth given that I was build manager for over a year before working in actual software development for my company's product (it's considered penance in my business ;). At that time Telelogic's predecessor-in-interest, Continuus, had not yet entered Australia and we were being supported by another firm. This firm was not very familiar with the product, and for many years the CCM expertise in my company exceeded that of the support firm in many areas. Some of my detailed knowledge may be out of date. I've been back in the software domain for a number of years.

CCM is built on an Informix database which contains objects, object attributes, and relationships. Above this level is the archive, which uses gzip to store object versions of binary types and a modified gnu rcs to store object versions of text types. Above this level is the cache, which contains extracted versions of all working-state objects and static (archived) objects in use within work areas. Working state objects only exist within the cache. The final level is the work area. Each user will have at least one, and that is where software is built. Under unix, the controlled files within the work area are usually symlinks to cache versions. Under windows, the controlled files must be copies. Object versions that are no longer in use can be removed from the cache by using an explicit cache clean command. A work area can be completely deleted at any time and recreated from cache and database information with the sync command. Atribitrary objects (including tasks and projects, which we'll get to shortly) can be transferred between CCM databases using the DCM object version transfer mechanism.

CCM is a task-based CM environment. That means that it distinguishes between the concept of a work area, and what is currently being worked on. The work area content is decided by the reconfigure activity which uses reconfigure properties on a project as its source data. A baseline project and a set of tasks to apply (including working state and "checked-in" (static) tasks). This set is usually determined by a set of task folders, which can be configured to match the content of arbitrary object queries.

Once the baseline project and set of tasks is determined by updating any folder content, the tasks themselves and the baseline project are examined. Each one is equivalent to a list of specific object versions. Starting at the root directory of the project, the most-recently-created version of that directory object within the task and baseline sets is selected. The directory itself specifies not object versions, but file-ids. The slots that these ids identify are filled out in the same way, by finding the most-recently-created version of the object within the task and baseline sets.

So, this allows you to be working on multiple tasks within the same work area. It allows you to pick up tasks that have been completed by other developers but not yet integrated into any baseline and include them in your work area for further changes. The final and perhaps most imporantant thing it allows you to do is perform a conflicts check.

The conflicts check is a more rigourous version of the reconfigure process. Instead of just selecting the most-recently-created object version for a particular slot, it actively searches the object history graph. This graph is maintained as "successor" relationships in the informix database. If the the graph analysis shows that any of the objects selected by the baseline or task set are not predecessors of the selected objects then a conflict is declared. The user typically resolves this conflict by performing a merge between the two selected but branch versions using a three-way diff tool. Conflicts are also declared if part of a task is included "accidentally" in a reconfigure. This can occur if you have a task A and task B where B builds on A. When B is included, but A is not included some of A's objects will be pulled into the reconfigure by virtue of being predecessors of "B" object versions. This is detected and the resolution is typically to either pull A in as well, or to remove B from the reconfigure properties.

The conflicts check is probably the most important feature of CCM from a user perspective. Not only can you see that someone else has clobbered the file you're working on, but you can see how it was clobbered and how you should fix it. On the other side, though, is the build manager perspective. Task-based CM makes the build manager role somewhat more flexible, if not actually easier.

The standard CCM model assumes you will have user work areas, an integration work area, and a software quality assurance work area. User work areas feed into integration on a continuous or daily basis, and every so often a cut of the integration work area is taken as a release candidate to be formally assessed in the slower-moving software quality assurance work area. Each fast moving work areas can use one of the slower-moving baselines as its baseline project (work area, baseline, and project are roughly interchangeable terms in CCM). Personally, I only used an SQA build within the last few months or weeks of a release. The means of delivering software to be tested by QA is usually a build, and you often don't need an explicit baseline to track what you gave them in earlier project phases.

One way we're using the CCM task and projects system at my place of employment is to delay integration of unreviewed changes. Review is probably the most useful method for validating design and code changes as they occur, whether it be document review or code review. Anything that hasn't been reviewed isn't worth its salt, yet. It certainly shouldn't be built on top of by other team members. So what we do is add an approved_by attribute to each task. While approved_by is None, it can be explicitly picked up by developers if they really need to build upon it before the review cycle is done... but it doesn't get into the integration build (it's excluded from the folder query). When review is done, the authority who accepts the change puts their name in the approved_by field, and either that person or the original developer does a final conflicts check and merge before the nightly build occurs. That means that work is not included until it is accepted, and not accepted until it passes the conflicts check (as well as other check such as developer testing rigour). In the mean-time other developers can work on it if they are prepared to have their own work depend on the acceptance of the earlier work. In fact, users can see and compare the content of all objects, even working state objects that have not yet been checked in. That's part of the beauty of the cache concept, and the idea of checking out objects (and having a new version number assigned to the new version) before working on them.

I should note a few final things before closing out this blog entry. Firstly, I do have to use a customised gnu make to ensure that changes to a work area symlink (ie, selection of a different file version) always cause a rebuild. It's only a one-line change, though. Also CCM is both a command-line utility and a graphical one. The graphical version makes merging an understanging of object histories much easier. There is also a set of java GUIs which I've never gotten around to trying. Telelogic's Change Synergy (a change request tracking system similar in scope to bugzilla) is designed to work with CCM, and should reach a reasonable level of capability in the next few months or years but is currently a bit snafued. Also, I haven't gone into any detail about the CCM object typing system or other aspects that there are probably better solutions to these days anyway. I also haven't covered project hierarchies, or controlled products which have a few interesting twists of their own.

Benjamin

Adrian Sutton writes about wanting to use tests as specifications for new work to be done:

You need to plan the set of features to include in a release ahead of time so you wind up with a whole heap of new unit tests which are acting as this specification that at some point in the future you will make pass.  At the moment though, those tests are okay to fail.

It isn't crazy do want to do this. It's part of the eXtreme Programming (XP) model for rapid development. In that model, documentation is thrown away at the end of a unit of work or not produced at all. The focus is on making the code tell the implementation story and making the tests tell the specification story. What the XP model would call what you're trying to do is not unit testing, but acceptance testing:

Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release.

Implicit in this model is the test score. In my line of work we call this the test report, and it must be produced at least once per release but preferrably once per build. A simple test report might be "all known tests pass". A more complicated one would list the available tests and their pass/fail status.

Adrain continues,

The solution I'm trying out is to create a list of tests that are okay to have fail for whatever reason and a custom ant task that reads that list and the JUnit reports and fails the build if any of the tests that should pass didn't but lets it pass even if some of the to do tests fail.

If you start from the assumption that you'll produce a test report the problem of changes to the set of passed or failed tests can become a configuration management one. If you commit the status of your last test report and perform a diff with the built one during the make process you can break the build on any unexpected behaviour in the test passes and fails. In addition to ensuring only known changes occur to the report it is possible to track (and review) positive and negative impacts on the report numbers. All the developer has to do is check in a new version of the report to acknowledge the effect their changes have had (they're triggered to do this by the build breakage). Reports can be produced one per module (or one per Makefile) for a fine-grained approach. As a bonus you get to see exactly which tests were added/removed/broken/fixed at exactly which time, by whom, and who put their name against acceptance of the changed test report and associated code. You have a complete history.

This approach can also benefit other things that are generated during the build process. Keep a controlled version of your object dump schema and forever after no unexpected or accidental changes will occur to the schema. Keep a controlled version of your last test output and you can put an even finer grain on the usual pass/fail criteria (sometimes it's important to know two lines in your output have swapped positions).

I know that the revision control system is one of the essential tools of a programmer, behind only the compiler/interpreter and the debugger, but so far for TransactionSafe I haven't used one. I suppose it is also telling that I haven't used a debugger so far, relying solely on the old fashioned print(f) statement to get to the bottom of my code's behaviour. Anwyay, on the issue of revision control I have been roughly rolling my own. I toyed with rcs while the program was small enough to fit reasonably into one file, but since it broke that boundary I have been rolling one tar file per release, then renaming my working directory to that of my next intended revision.

That is a system that works well with one person (as I remain on this project) but doesn't scale well. The other thing it doesn't handle well is the expansion of a project release into something that encapsulates more than a few small changes. A good revision control system allows you to roll back or review the state of things at a sub-release level. To use the terminology of Telelogic CM Synergy, a good revision control system allows you to keep track of your changes at the task level.

Well, perhaps by now you've worked out why I haven't committed to using a specific tool in this area yet. I use CM Synergy professionally, and am a little set in my ways. I like task-based CM. In this model (again using the CM Synergy terminology), you have a project. The project contains objects from the central repositry, each essentially given a unique object id and a version number. A project version can either be a baseline version of an earlier release, or a moving target that incorporates changes going forward for a future release. Each developer has their own version of the project, which only picks up new version as they require it. Each project only picks up changes based on the developer's own criteria.

The mechanism of change acceptance is the reconfigure. A reconfigure accepts a baseline project with its root directory object, and a list of tasks derived by specific inclusion or by query. Tasks are beasts like projects. They each contain a list of specific object versions. Directories contain a specific list of objects, but does not specify their version. The reconfigure process is simple. Take the top-level directory, and select the latest version of it from the available tasks. Take each of its child objects and choose versions of them using the same algorithm. Recurse until done. Developer project versions can be reconfigured, and build manager project versions can be reconfigured. It's easy as pie to put tasks in and to pull them out. Create a merge task to integrate changes together, and Bob's your uncle.

This simple mechanism, combined with some useful conflicts checking algorithms to check for required merges make CM Synergy the best change management system I've used. It's a pity, really, because the interface sucks and its obvious that no serious money has gone into improving the system for a decade. Since its closed source I can't make use of it for my own work, nor can it be effectively improved without serious consulting money changing hands.

I've been out of the revision control circles for a few years, now, but Anthony Towns' series of entries regarding darcs has piqued my interest. Like CM synergy, every developer gets to work with their own branch version of the project. Like CM synergy, each unit of change can be bundled up into a single object and either applied or backed out of any individual or build manager's project. The main difference between darcs' and CM Synergy's approaches appear to be that while CM Synergy's change unit is a set of specificially selected object versions, darcs change unit is the difference between those object versions and the previous versions.

It is an interesting distinction. On the plus side for darcs, this means you don't have to have explicit well-known version numbering for objects. In fact, it is likely you'll be able to apply the patch to objects not exactly the same as those the patch was originally derived-from. That at least appears to bode well for ad-hoc distributed development. On the other side, I think this probably means that the clever conflicts checking algorithms that CM Synergy uses can't be applied to darcs. It might make it harder to be able to give such guarantees as "All of the object versions in my release have had proper review done on merges". Perhaps there are clever ways to do this that I haven't thought of, yet.

On the whole darcs looks like the revision control system most attuned to my way of thinking about revision control at the moment. I'll use it for the next few TransactionSafe versions and see how I go.

While I'm here, I might as well give an impromptu review of the telelogic products on display: