Migrations to Git and GitHub continue

Last year I started migrating from CVS to Git and GitHub (see First migrations from CVS to Git), which is something I was willing to start back in 2018 (see Migration to Git and GitHub), but planned since even longer ago. Well, it proved to be a tough task, because I'm still migrating as many of my CVS repositories required corrections, so the history could be properly transferred in Git. During the migration I was surprised how negligent I was to my sources, because I found some uncommitted changes from as far back as 10 years ago, corrupted revision control files and various history inconsistencies. After migrating some projects, I immediately started working on updates and fixes, so I already continue my work on GitHub. I also enabled continuous integration for some projects using Travis CI and GitHub Actions.

I'm currently migrating my Slackware package's build scripts, which is a collection of over 300 Shell scripts and related files organized as separate repositories in dedicated directories, but under the same root. In CVS this was completely right, but it does not produce good history in Git, because the version tags or branches (e.g. FFmpeg-3_4_7 or MySQL-5_5) are specific only to some build script and used only by the files in its respective directory. I thus decided to migrate these separately and then combine them with another repository with sub modules. There were many problems with this migration as well.

These were some the problems:

• tags or branches with similar names (e.g. TEST-123 and TEST_123). Those are easy to correct - I just delete the wrong tag/branch;
• misplaced tags or branches. Since in CVS I was tagging and branching as necessary some tags and branches were put on different files at different time, which in Git history resulted in commits with the message "This commit was manufactured by cvs2git to create tag 'TEST-123'" or "This commit was manufactured by cvs2git to create branch 'TEST-123'" with files being added, deleted or modified to adjust history for the tag or branch respectively. I'm fixing such problems by reordering the problematic commits, which is easy by just changing the time in the revision control file in the repository, but requires time to review and understand the reason;
• unused files not deleted in version history. This also produced the commits with messages mentioned in the previous bullet point. I'm fixing these by deleting the unused files with past date, so such files are not considered by consecutive commits;
• files belonging to a branch, but committed to main trunk instead. It's similar to the above. In some cases, there are files that should exist only on some branch (e.g. like patches to fix issues with specific software versions), but were committed to main trunk. As in CVS there is no big difference between tags and branches it was enough just to tag the changes for checking out properly. I'm moving these from main trunk into a branch, which means deleting revisions from main trunk and moving them on the branch;
• spelling errors. Since I mostly committed to CVS on the command line and I did not use spell checker there were many commit messages with spelling errors that irritate my eyes. I'm correcting them after initial migration to Git, so I could check all messages at once.

Two utilities proved to be of great help after migration - licensee and github-linguist. I use the first to check whether license is detected properly. I had some problems (see issues 361 and 392) with some of my projects, so I now check before pushing to GitHub. The second is useful for fine tunning languages detection (e.g. I'd like to see just SQL and not PLSQL, PLpgSQL, SQLPL or TSQL that linguist would detect although all my published SQL sources are for MySQL. And for me SQL is code, not data).

Anyway, this post is to note that my migrations to Git continue. I've made great progress last year by migrating 28 repositories. And this year by now I migrated 114 more. So hopefully by the end of the year I'll be free from using CVS (or at least only by exception). Initially I planned migrating everything, but now I consider skipping some low value projects or examples, which if necessary, I could migrate later.

First migrations from CVS to Git

Last May I had the good intention to start gradually migrating my old CVS repositories to Git, but I only managed to begin the process this weekend. Before plunging deeper I decided to migrate some smaller repositories with linear history, so I could exercise, verify and master the process. As a result some small projects already landed on my GitHub profile (see guthub.com/gdsotirov). I have read much about migrating from CVS and SVN to Mercurial and Git in relation to an internal project in the company I'm working for. I have also already experimented migrating from CVS to SVN in the past and from CVS to Hg/Git more recently, so I considered myself prepared, but still there are some things to consider when doing such migration.

Repository preparation

Revision log messages. You may have a convention for writing revision log messages in CVS, but I did not have one for my personal projects, so I have written comments on single line and on multiple lines as unnumbered list like this:

* Add this
* Modify that

I had to modify these comments in CVS, before the migration, because these simply do not look good in TortoiseGit and GitHub. I also aligned some similar comments on files committed in few minutes time span, so that the migration could group them in a single revision. Depending on whether you had a convention for the format of the revision log messages in CVS you may need to make other adjustments. The easiest way in such case may be to modify directly the revision control files with perl, sed, awk, etc. while of course paying caution not to ruin your history (i.e. such operations should not be performed on "live" production repository).

Authors. You should be able to identify all your authors, because in CVS repositories they are normally UNIX login names and the users may no longer exist in the system. I've seen even cases where the same username was reused by different developers having the same given and family names. It's hard to identify and fix such revisions unless you're aware of developers' internship in the company, but what if their time in the company overlaps?
Hopefully, I do not have this problem with my personal repositories, but still I had to rewrite my username as full name and email address (see more about this in chapter Step One: The Author Map of Eric Raymond's DVCS migration HOWTO). It's very important to rewrite the authors, because otherwise the commits won't be properly identified in GitHub (see Contributions that are counted). If you need to fix usernames it's preferable to do it in the CVS repository before the conversion. And the easiest way to map usernames to authors is to rewrite the converted Git repository as I write about below.

Tools

In such a process one of the most important things is to chose the right tools, because you simply cannot go without them. Last year at work we evaluated several options and finally choose cvs2git, which is part of cvs2svn project. The command is written in Python and supports various options. For me it is important to make the best possible migration, so I had to make the following modifications in the example options file:

KeywordHandlingPropertySetter('expanded'),

from cvs2svn_lib.keyword_expander import _KeywordExpander
# Ensure dates are expanded as YYYY/MM/DD as in CVS 1.11
_KeywordExpander.use_old_date_format()

The first line ensures that CVS keywords (e.g. $Id$, $Date$, etc.) are expanded in text files (see topic cvs2svn changes 'date' string in source codes). The following three lines ensure that dates in keywords are expanded as in CVS 1.11 or in YYYY/DD/MM format, instead of the default format YYYY-MM-DD as in CVS 1.12 and SVN.

Migration

The way to covert a CVS repository to Git is described in cvs2git's documentation (see chapter Usage), so just read it. I wrote myself a shell script to execute the different steps defined there. Initially I used cvs2git only with command line options, but because of my requirements for keywords expansion (see above), I had to improve the script to generate options file and use it instead. Like this I retained the way I call the script by just providing path to the CVS repository.

For now I haven't had problems with the migration - the history was represented properly in Git. However, with many tags and branches over different files things could get really messy as I found out with the internal project at work. After the migration I check the result and if necessary fix more revision log messages. Then, I just rewrite authors with the script given in Changing author info and finally import the repository to GitHub (e.g. I use git daemon --verbose --export-all --base-path="/path/to/repo" to serve the converted repository). After the import to GitHub I have to add LICENSE and README.md files, but this is quite trivial directly through the web interface.

And that's all. I have more repositories to migrate and I really hope I'll be able to finish the task until the end of the year, but it would all depends on how quickly I'll be able to prepare the repositories, because after this it's the jobs of the tools.

Migration to Git and GitHub

Several years ago I decided to migrate all my personal projects to a distributed version control system (DVCS), but simply didn't found the time. Nowadays Git is de facto standard for version control, so I decided to migrate to Git and put all my personal projects to GitHub. So this year I would be gradually migrating my projects, starting with the least active ones. Most of them are still on CVS, which makes the migration more complex given CVS and Git are quite different in many aspects, but also because for me it's important to keep the history of my modifications (i.e. I do not want just to copy the source and initialize and new Git repository inside). Wish me luck and cross your fingers for me ;-)