Git 3.0's upcoming SHA-256 default will be a costly mistake
blog.gitbutler.com
Git 3.0 will make SHA-256 the new default content hashing algorithm and it will be an incomprehensibly expensive and ultimately valueless and avoidable global nightmare.
  • lobsters__interlandi@crustaceanson.theshelf.topM
    0·
    1 month ago

    I am no cryptographer, nor an expert on the “public health” side of software engineering, but I would wager that this decision is supposed to be forward-looking rather that trying to mitigate some non-existent immediate issue. I do not like that the article positions the change as the latter (to an extent, I know they offered more than that). 15 years from now, some number of the world’s repositories will still be using SHA-1. Nobody can say for certainty that SHA-1 will not be broken in 15 years. That number will be smaller then, given that they are making the switch now, than if Git had waited. One thing I would also add is that adding additional hash functions on top of existing ones may not be suitable solution, since, at least intuitively, it must increase both the cryptographic integrity and the vulnerable surface simultaneously.

    score 1 · canonical

    • lobsters__siru@crustaceanson.theshelf.topM
      0·
      1 month ago

      I agree with you. The article seems to pose this as a “we can avoid this pain by simply remaining on sha1” while in truth this seems like it would end up putting us in a worse situation 20 years into the future if sha1 gets more broken and there is even more legacy code still dependent on sha1 references remaining untouched. This feels a bit like setting ourselves up for a Y2K-like event if the new sha256 doesn’t start slowly being rolled out now.

      score 1 · canonical

        • lobsters__fanf@crustaceanson.theshelf.topM
          0·
          1 month ago

          SHA-1 and SHA-2 are fairly similar designs, so any attack on SHA-2 is also likely to affect SHA-1. This similarity is why there was a hash function competition to come up with a successor SHA-3 with a different design. Even so, SHA-2 appears to be solid.

          Too Much Crypto summarizes the situation in panglossian style:

          • SHA-1: The collision found in 2017 uses a refined version of the 2005 collision attack, and ran with a complexity close to the one estimated in 2005. The 2019 chosen-prefix attack is a notable improvement and is estimated to run with similar complexity.

          • SHA-256: Cryptanalysts seem to have given up on looking for collision attacks. The much higher non-linearity, compared to SHA-1, is indeed intimidating and unlikely (in our opinion) to be worked around.

          score 1 · canonical

  • lobsters__hailey@crustaceanson.theshelf.topM
    0·
    1 month ago

    There’s one glaring hole in Scott’s “trust is in the distribution” argument specifically as it relates to GitHub which is that GitHub commingles objects across the whole fork network. Only the refs namespace is separated.

    You don’t need to breach GitHub’s authentication to get a forged object into rust-lang/rust. It is much simpler - you fork the repo and push the forged object to your fork. It is then visible in the parent.

    I am actually quite surprised that the article overlooks this angle given that Scott was a cofounder of GitHub.

    score 1 · canonical

      • lobsters__mort@crustaceanson.theshelf.topOPM
        0·
        1 month ago

        SHA-1 is semi-broken. There have been collisions found and there have been found cryptographic weaknesses which makes it significantly weaker than the number of bits would suggest. But it’s still not by any means easy to forge a commit which both has the malicious payload you want and has a hash collision with a relevant commit. The collisions which have been made so far have very much been proof-of-concept.

        But, like with post-quantum encryption algorithms, it’s probably better to transition before it becomes a big enough problem that there are exploits in the wild.

        Also, Git and Git hosts today use what they call “sha1dc”; sha1 with collision detection. I don’t understand how the cryptography works but it apparently is able to somewhat work around the specific issues which have been used to produce collisions so far. So pushing e.g two colliding PDFs to GitHub just won’t work these days.

        score 2 · canonical

  • lobsters__df@crustaceanson.theshelf.topM
    0·
    1 month ago

    I had this same experience working on a proprietary backup system that used hashes heavily in 2016 or so. “FIPS says we can only use SHA-256.” “But we use hashes for deduplication, not security.” “Doesn’t matter, FIPS says it.” “But we have thousands of users. Many are on older CPUs where that hash is expensive.” “Doesn’t matter.” “Can we just throw truncated SHA-256 in the same field as SHA-1 and call it a day?” “No.” Anyway we upgraded all our users to SHA-256.

    score 1 · canonical

    • lobsters__jlaverack@crustaceanson.theshelf.topM
      1·
      1 month ago

      I’ve found a lot of the time that FIPS builds just straight-up don’t include “unapproved” hashing algorithms in libraries. Even if you could argue an exception for SHA-1 in some non-security case, the software build may simply not include the functions to do it at all.

      I’ve hit this problem with the Linux kernel before, where a FIPS=1 kernel will not load many crypto modules and if you were relying on the kernel implementations for MD5 or SHA-1 then you’re out of luck.

      score 1 · canonical

    • lobsters__adrien@crustaceanson.theshelf.topM
      0·
      1 month ago

      My understanding for SHA-1 in FIPS is that it gets a harsher treatment than MD5 did.

      MD5 got a transition and accepted usecases. SHA-1 on the other hand basically disappears. You could say that if it doesn’t exist for FIPS then you can use it for what isn’t regulated by FIPS but you should probably get ready to pay lawyers and have a lot of uncertainty around your product (good luck selling it in contracts that need years or decades of support). I think your example is an example of that: you were able to do the technical change while noone got through the justification approach.

      I think that part of that is the past experience with MD5 and the difficulty with assessing usage of given codes. It’s quite easy to detect non-obfuscated/adversarial cryptography code and terribly difficult to asses how and when it’s used. The more realistic option is to remove SHA-1 code completely. I think that’s the approach in FIPS.

      I don’t understand the companies (including gitbutler) that try to avoid the change because they’re going to aleniate a lot of customers. They don’t even have to kill SHA-1 support in their current stuff: they need to have the compatibility for SHA-256 and the ability to skip SHA-1. These are very common developments when you have a mix of US-government and non-US-government customers.

      score 1 · canonical

    • lobsters__fanf@crustaceanson.theshelf.topM
      0·
      1 month ago

      I think that reasoning led to a svn repository being catastrophically corrupted by the shattered sha-1 collision.

      It’s a common fallacy that deduplication doesn’t depend on the security properties of cryptographic hash functions: if the system doesn’t have any extra machinery to handle a collision then a broken hash function will cause loss of data integrity and probably availability, two of the three legs of the CIA triad.

      score 1 · canonical

    • lobsters__mort@crustaceanson.theshelf.topOPM
      0·
      1 month ago

      How can you use SHA-1 for deduplication in a way that’s not somewhat security-related? If you do file-level deduplication, and someone has a file A backed up, then downloads a file B which collides with A, then restores to a point where they had B on their system, won’t your back-up system have replaced B with A?

      Or, to invert it: if I get someone to download file B, won’t I have poisoned their system so that if they ever download A, it gets replaced by my file B upon backup restore?

      This seems like an avenue for security issues. Sure, people shouldn’t download files from sources they don’t trust… but also, people don’t expect that merely downloading a file will permanently poison their back-ups.

      score 1 · canonical

      • lobsters__majaha@crustaceanson.theshelf.top
        0·
        1 month ago

        Well, according to the article, it’s not feasible to create a collision for a file someone already has. Only to create two files that collide with each other. So an attacker would have to make you download two files, which might swap each others contents (although I suppose it’s possible that this could break your backup system, as it’s unexpected). This doesn’t seem that scary to me. This turns the case where they’ve given you malware, into a case where they’ve given you malware that might activate only after a restore.

        score 1 · canonical

        • lobsters__moltonel@crustaceanson.theshelf.topM
          0·
          1 month ago

          It’s not too hard to predict the content of some files. For example I could get my malware to hash the same as the GPL license text. Or I could email you some harmless content, which hash-collides with malware that’s already in your backup system. If github (which shares git refs across repos) didn’t have collision protection, I could fork you project, push my malware to my fork, then send you a clean-looking PR.

          score 1 · canonical

          • lobsters__fanf@crustaceanson.theshelf.topM
            0·
            1 month ago

            You are proposing a second preimage attack (generate a file (called “the preimage of the hash” in the jargon) with a predetermined hash), not a collision attack (generate two files with any hash so long as they hash the same).

            A second preimage attack’s expected cost for a secure hash is the full size of the hash: 2^256 trial hashes for SHA-256 or 2^160 trials for SHA-1. (Or to be precise, 2^255 and 2^159 because you expect to find a match before testing every possibility.)

            A collision attack benefits from the birthday paradox so its expected cost is the square root, or half the exponent, so 2^128 trials for SHA256 or 2^80 trials for SHA-1. The standard security level for cryptographic operations is 2^128 symmetric ops, which SHA256 passes but SHA-1 does not.

            The weaknesses in SHA-1 reduce a collision attack from 2^80 to about 2^61 trials, which is feasible to carry out in a reasonable length of time.

            SHA-1 is not vulnerable to second preimage attacks at least partly because squaring the number of trials makes the attack unfeasible even for a weak hash. Even MD5 is not vulnerable to second preimage attacks.

            score 3 · canonical

            • lobsters__interlandi@crustaceanson.theshelf.topM
              0·
              1 month ago

              Is the nature of SHA-1, or any commonly used cryptographically secure hash function, that it is not any harder to just put the random bytes necessary to make some malicious file collide with a benign one anywhere you want than it would be to construct an meaningless string from scratch that collides? Like could I put them in some unused segment of an ELF binary, or something? That’s not a rhetorical question, I’m just curious.

              score 1 · canonical

              • lobsters__fanf@crustaceanson.theshelf.topM
                1·
                1 month ago

                There are subclasses of collision attacks that depend on details of how the collision is constructed.

                Merkle-Damgård hash functions (like MD5, SHA-1, SHA-2) are subject to length extension because the hash value exposes the entire state of the function, so once you know h(M) you can calculate h(M++X) for any suffix X of your choice, and if you have a collision h(M) == h(M’) you can also calculate h(M++X) and h(M’++X). Depending on the details of the file format this can make collisions more powerful, because the colliding prefix can select different parts of the suffix so a single collision calculation can be reused for lots of different versions of an attack. The shattered collision used this in PDF, where the suffix X contained both the benign and malicious payloads, and the colliding prefixes M and M’ chose which payload to display.

                In a collision attack, M is typically split into a prefix P and some carefully-constructed blocks C that cause the collision. The C blocks are a random mess. There are roughly three classes of collision attacks depending on the structures of M==P++C and M’==P’++C’:

                P==P’ == “” - classical collision attack which in practice does not allow enough control over the structure of the message to create interesting attacks.

                P==P’ != “” - this is a “free start” collision attack, which allows the attacker to construct a prefix of a valid file in some format that can be used for length extension attacks, as in the shattered collision.

                P != P’ - a chosen-prefix attack, which allows two different files to be extended in a way that causes them to collide, almost as powerful as a second-preimage collision.

                score 1 · canonical

  • lobsters__grahamc@crustaceanson.theshelf.topM
    0·
    1 month ago

    Maybe it is my perpetual optimism, but I’m pretty convinced that as soon as it exists there will be incredible “market pressure” (from users, volunteers, customers, etc.) to make the integration and transition straightforward.

    I’m sure it will have bumpy and rocky periods, but I think we’ll get there more quickly than this article suggests.

    score 1 · canonical

    • lobsters__pzel@crustaceanson.theshelf.topM
      0·
      1 month ago

      Funny, I have the opposite gut feeling. There will certainly be “market pressure”, just as there was with the python2->3 transition, and yet despite the pressure, it was a protracted trauma for many organizations. The system on hand is not the big question, but, as some others have noted in other threads, it’s the huge ecosystem around the main system that is more thorny. cf. python the language vs. python-the-ecosystem.

      score 1 · canonical

      • lobsters__0x2ba22e11@crustaceanson.theshelf.topM
        1·
        1 month ago

        Good news: it can’t be nearly as bad as the py2 → py3 transition was. Even if we all decided to switch to a completely new version control system like Pijul or Mercurial or Darcs or something, it wouldn’t be nearly that bad. We already did it multiple times (e.g. mass exodus from svn to various DVCSes, then to git) and it was never a huge deal.

        You can convert one repo at a time. The vast majority of software builds don’t interact with the VCS tools of their dependencies other than “how do I download the source code given a URL”. Build tools can add support for new VCS types without breaking existing users of the existing ones.

        The python 2→3 transition had exceptionally nasty network effects: users couldn’t move to py3 until all their dependencies were available on it. The big frameworks users depended on couldn’t simply move to py3 unless most of their users had moved: they had to stay py2 compatible. For a few years (until python 3.2 was released) there wasn’t any good way of writing python code that would parse and function identically in both py2 and py3. (The 2to3 translation tools were imperfect and in practice everyone needed a way to write polyglot code instead.)

        score 1 · canonical