Is there any specific tooling you’re using for this endeavour? There’s a big tote of drives sitting on the floor here next to me that deserve the same treatment…
btrfs deduplication helps cut down on intermediate storage needs while I get all the file trees in one place so I can compare them.
Comparison is tricky since for trees evolve over time. I need to detect when two folders at arbitrary places in the file tree have highly similar content. I’ll roll up arrays of rapidhash values and throw each hierarchy level at Levenstein and see how far that gets me.
But I’m also going to try directly accessing the btrfs block identity data instead of doing a manual scan.
Dope! Yeah that’s going to be the big thing for me: drives from 5 years ago are going to have some content that isn’t on my current drives and some content that is a git repo that I do have on my current drives but not with the 5-year-old checkout. At least for those kinds of things I should be able to check the remotes and see if they match
I’m in this picture and don’t like it :)
Is there any specific tooling you’re using for this endeavour? There’s a big tote of drives sitting on the floor here next to me that deserve the same treatment…
score 1 · canonical
btrfs deduplication helps cut down on intermediate storage needs while I get all the file trees in one place so I can compare them.
Comparison is tricky since for trees evolve over time. I need to detect when two folders at arbitrary places in the file tree have highly similar content. I’ll roll up arrays of rapidhash values and throw each hierarchy level at Levenstein and see how far that gets me.
But I’m also going to try directly accessing the btrfs block identity data instead of doing a manual scan.
score 2 · canonical
Dope! Yeah that’s going to be the big thing for me: drives from 5 years ago are going to have some content that isn’t on my current drives and some content that is a git repo that I do have on my current drives but not with the 5-year-old checkout. At least for those kinds of things I should be able to check the remotes and see if they match
score 1 · canonical