A lot of people have been commenting on this over time so I figured I would put together a little post summarizing my thoughts on the matter and everything else. The post is going to be quite large. Every section can be read more or less separately, but they are all connected to each other.
I run a Chimera mirror at https://au.mirror.7bit.org/. To ensure atomic updates I use the atomic-rsync script, which is included in Rsync tarballs, but is not currently part of the Chimera rsync package (the mirror server runs Chimera).
atmoic-rsync manages a pair of directories, syncing into the inactive one, then swapping a symlink when the sync is done so what’s returned by the web server switches instantly. atomic-rsync uses the --link-dest option to populate the tree being synced from the currently active one to avoid storing duplicates of unchanged data.
This has worked well, although occasionally rsync would get stuck, or take a long time and the syncs would start overlapping. That was solved by invoking it with flock. My full cron line is:
I run a Chimera mirror at https://au.mirror.7bit.org/. To ensure atomic updates I use the atomic-rsync script, which is included in Rsync tarballs, but is not currently part of the Chimera rsync package (the mirror server runs Chimera).
atmoic-rsyncmanages a pair of directories, syncing into the inactive one, then swapping a symlink when the sync is done so what’s returned by the web server switches instantly.atomic-rsyncuses the--link-destoption to populate the tree being synced from the currently active one to avoid storing duplicates of unchanged data.This has worked well, although occasionally
rsyncwould get stuck, or take a long time and the syncs would start overlapping. That was solved by invoking it withflock. My full cron line is:When I set this up I wasn’t aware of the mechanism q66 used on the Buildbot workers, if I’d known that then I could have done the same.
score 3 · canonical