Magma Backup and Recovery
Last updated at 4:37 pm UTC on 23 September 2007
Background
Every successful commit is numbered consecutively by the server. Each CommitPackage of each commit is serialized and appended to a log file known as "commits.log". Updates to the various repository files are flushed to disk only every five seconds, but several commits could have happened during that time. If the server is forcibly terminated by an outside event (such as a power outage), the commits.log file is read upon the next startup to replay those last few commits to the repository files.
In this way, any client that receives a successful response from a commit request can be assured their changes are secured in the database.
To manage the size of the commits.log file, every five minutes or, as defined by #logStartFrequency, it is safely closed and renamed into a subdirectory within the repository directory called "commits". Its name is also changed within this subdirectory to "commits.n.log", where n is the number of the highest commit in that file. The "commits.log" file, in the main repository directory, is known as the "active" commit log file. The old renamed files, in the commits subdirectory, are known as "archive" commit logs.
The active commit log file may also be archived on demand, without waiting for the (five minute) timer to expire by:
myRepositoryController archiveCommitLog
These old commit log files are generally of no use to the repository that wrote them, because it would only ever need up to the last five seconds worth of data in the active log. The old files are useful, however, for maintaining a warm backup on separate hardware, continuously replaying the files and providing a read-only copy of the main repository, alleviating considerable burden imposed by query and reporting functions (where up-to-the-second is not necessary) from the main repository. This read-only copy could also be ready to become the main repository in the event of a failure of the main.
Making a Backup
Backups can be taken while the server is running, with no interruption of service.
myRepositoryController fullBackupTo: aFileDirectory
aFileDirectory may be a FileDirectory or a ServerDirectory (for backing up straight to an FTP server).
aFileDirectory is created if it doesn't already exist. If a repository already exists at that location, a warning will be issued before deleting all files in the directory. Proceed past the warning and the active commit log is archived before a background (Smalltalk) Process is forked to copy the repository files. To ensure consistency, all flushing to the repository files from memory is suspended during this copy creating a potential increase in memory utilization if activity is high.
Once the files are copied, a bit is finally set in the target repository indicating it is in "restore mode," meaning it can now replay commit log files beginning with the next higher commit number. This is discussed in detail in the subsequent section, "Restoring from a Backup".
The backup can now be started as a separate, read-only repository. In restore mode, it can apply commit logs from the original repository.
Restoring from a Backup
When a backup is taken, it is a snapshot of the persistent model up to a certain commit, identifiable by the #commitNumber of the repository. While the files are being copied, further commits continue to be received and written to the active commit.log file, but not the repository files because they are being copied. Therefore, the copied repository files are behind even before they are finished being written. To bring them up to date, the commit log files must be placed in the backup repository's "commits" subdirectory where they will be found when the repository is started and automatically replayed.
Restoring to a Point in Time
In addition to the commitNumber, Magma also stamps each record in the commit logs with the local DateAndTime.
In (hopefully) rare cases, an application bug or user error may result in an unintended commit with drastic consequences to the persistent model. If you know the approximate time the unintended commit occurred, a drastic solution is available; recover to a point in time.
In this case, a prior backup prior to the drastic event must be available. Put the commit logs timestamped up to and, even, a little beyond the drastic commit into the "commits" subdirectory.
When Magma starts, it will determine that the repository is behind the commit logs and, therefore, a forward recovery is needed. A MagmaForwardRecoveryNeeded warning message will be displayed:
"A forward recovery from 2007-08-02T22:01:09+00:00 is necessary. If you wish to recover to a point-in-time, press Debug and save the desired target DateAndTime in my 'targetRecoveryTime'."
Warm Backups
As mentioned, the backup is a nearly-exact copy of the repository. It differs by one bit from the original, its #restoreMode bit is set. When a repository is in this mode commits are not permitted because it expects to replay commits that occurred since the backup.
To do this, simply open the repository:
myRepositoryController := MagmaRepositoryController open: (MagmaLocalLocation path: '/path/to/your/repository')
Proceed past the warning and recovery to the highest available commit-log occurs automatically. Once up to date, future commit logs may be replayed via the #replayCommitLogs message on the already-open repository:
myRepositoryController replayCommitLogs
In this manner, an up-to-the-minute warm backup can be maintained.
Failing-over to the Warm Backup
As mentioned, backup repositorys have their restoreMode bit set, but may be brought on-line normally in read-only mode. When in restoreMode, the repository may only advance forward by applying commit.log files.
In exceptional cases it will be necessary to resume with the backup repository. For commits to be permitted on the warm backup, #commitRestore must be executed:
myRepositoryController commitRestore
Once this is performed, the repository now accepts its own commits, and can no longer accept commit logs from any other repository. It should be considered the "new master" and backed up right away and a new warm backup established on separate hardware.
Cleaning Up Old Commit Logs
Because of the variety of possible configurations, there is no way Magma can make always correct assumptions about when to clean up old commit logs. They will continue to accumulate in the "commits" directory until you ask your repositoryController to clean them up:
myRepositoryController deleteUnnecessaryCommitLogFiles