My emails forced me to, locking me out of accounts I needed to access.

Microsoft had me fill this form, to “prove” I was the rightful owner of the account, after some suspicious login-attempts from an African country. The form included fields like: name (I don’t think I supplied at creation, or a false one), other email addresses, previous passwords (which potentially yield completely unrelated passwords), etc.; only for the application to be rejected and locking me out of my primary email for a full month. After that outright violation, I immediately switched to Disroot, and haven’t had any of said problems ever since. I backup all its contents locally using Thunderbird, and delete the origins from the server afterwards.

Many platforms have this messed up dark pattern, of revoking one’s access to a real-world dependencies, unless giving in to the service’s demands. Enforcement of 2FA is another one of those “excuses” for this type of misbehavior, and so is bot-detection.

Yeah, I think they employ a pretty sophisticated bot detection algorithm. I vaguely remember there being this ‘make 5 friends’ objective, or something along those lines, which I had no intention of fulfilling. If a new account, having triggered the manual reviewing process, doesn’t adhere to common usage patterns, simply have them supply additional information. Any collateral damage simply means additional data, to be appended to Facebook’s self-profiling platform… I mean, what else would one expect when Facebook’s first outside investor was Palentir’s Peter Thiel?

It’s been like that for quite a while. I remember deleting all big-tech accounts in 2019, and shortly after, Facebook started requiring login for full public page access. Therefore I created a burner account using a ‘this person does not exist’ picture, which provided me short-lived access after manual review. For account recovery, I was required to supply additional selfies (or even video-selfies?), but at that point I gave up.

Well, we are using them today for human programmers, so… :-)

True that haha…

I don’t know: it’s not just the outputs posing a risk, but also the tools themselves. The stacking of technology can only increase attack-surface it seems, at least to me. The fact that these models seem to auto-fill API values, without user-interaction, is quite unacceptable to me; it shouldn’t require additional tools, checking for such common flaws.

Perhaps AI tools in professional contexts, can be best seen as template search tools. Describe the desired template, and it simply provides the template, it believes most closely matches the prompt. The professional can then “simply” refine the template, to match it to set specifications. Or perhaps rather use it as inspiration and start fresh, and not end up spending additional time resolving flaws.

No worries! :)

The main paradox here, seems to be: the 70% boilerplate head-start being perceived faster, but the remaining 30% of fixing the AI-introduced mess, negating the marketed time-savings; or even leading to outright counterproductivity. At least in more demanding environments, not cherry picked by the industry, shoveling the tools.

I understand you’ve read the comment as a single thing, mainly because it is. However, the BLE part is an additional piece of critique, which is not directly related to this specific exploit; neither is the tangent on the headphone jack “substitution”. It’s, indeed, this fast pairing feature, which is the subject of the discussed exploit; so you understood that correctly (or I misunderstood it too…).

I’m however of the opinion, BLE being a major attack vector, by design. These are IoT devices that, especially when “find my device” is enabled (which in many cases isn’t even optional: “turned off” iPhones for example), do announce themselves periodically to the surrounding mesh, allowing for the precise location of these devices; and therefore also the persons carrying them. If bad actors gain access, to for example Google’s Sensorvault (legally in the case of state-actors), or would find ways of building such databases themselves; then I’d argue you’re in serious waters. Is it a convenient feature, to help one relocate lost devices? Yes. But this nice-to-have, also comes with this serious downside, which I believe doesn’t even near justify the means. Rob Braxman has a decent video about the subject if you’re interested.

It’s not even a case of kids not wanting to switch, most devices don’t even come with 3.5mm jack connectors anymore…

If the devices weren’t previously linked to a Google account … then a hacker could … also link it to their Google account.

This already severely limits the pool of potential victims; but still a more practical exploit indeed. It’s almost as if this BLE tracking is a feature, rather than an exploit. And if you want to be notified of a device following you around, one has to perpetually enable BLE on their smartphone. But of course, headphone jacks are a thing of the past, and wireless is clearly the future. :)

But you need to be in close proximity (~15m max) to stalk a victim? You might as well just follow them around physically then. Perhaps when the victim is in a private location, eavesdrop on their conversation or locating their position within there, might be a possibility. But ear raping would, of course, constitute the most significant danger of all. Also WhisperPair, not WhisPair?

AI reviews don’t replace maintainer code review, nor do they relieve maintainers from their due diligence.

I can’t help but to always be a bit skeptical, when reading something like this. To me it’s akin to having to do calculations manually, but there’s a calculator right beside you. For now, the technology might not yet be considered sufficiently trustworthy, but what if the clanker starts spitting out conclusions, which equal a maintainer’s, like 99% of the time? Wouldn’t (partial) automation of the process become extremely tempting, especially when the stack of pull request starts piling up (because of vibecoding)?

Such a policy would be near-impossible to enforce anyway. In fact, we’d rather have them transparently disclose the use of AI than hide it and submit the code against our terms. According to our policy, any significant use of AI in a pull request must be disclosed and labelled.

And how exactly do you enforce that? It seems like you’re just shifting the problem.

Certain more esoteric concerns about AI code being somehow inherently inferior to “real code” are not based in reality.

I mean, there’s hallucination concerns, there’s licensing conflicts. Sure, people can also copy code from other projects with incompatible licenses, but someone without programming experience is less likely to do so, than when vibecoding with a tool directly trained on such material.

Malicious and deceptive LLMs are absolutely conceivable, but that would bring us back to the saboteur.

If Microsoft itself, would be the saboteur, you’d be fucked. They know the maintainers, because GitHub is Microsoft property, and so is the proprietary AI model, directly implemented in the toolchain. A malicious version of Copilot could, hypothetically, be supplied to maintainers, specifically targeting this exploit. Microsoft is NOT your friend, it closely works together with government organizations; which are increasingly interested in compromising consumer privacy.

For now, I do believe this to be a sane approach to AI usage, and believe developers to have the freedom to choose their preferred environment. But the active usage of such tools, does come with a (healthy) dose of critique, especially with regards to privacy-oriented pieces of software; a field where AI has generally been rather invasive.

Yes, because they constitute a significant portion, of the eyes, traditionally involved with doing the verification of software. You can allow a potentially cherry-picked group of researchers to do the verification, on behalf of the user-base, but that hinges on a “trust me bro” basis. I appreciate you’ve looked into the process in practice, but please understand that these pieces of software, are anything but simple. Also if a state-actor were to deliberately implement an exploit, it wouldn’t be necessarily obvious at all, even if source-code was available; they’re state-backed, top of their game security-researchers themselves. Even higher tier consumer-grade computer viruses, won’t execute in a virtualized environment, precisely to avoid being detected. They won’t compromise when unnecessary, and might only be exploited when absolutely required; again to avoid suspicion.

I fully agree with the last paragraph though, and believe there to be an overreliance on digital systems over all. In terms of FOSS software, you have to rely on many, many different contributors to facilitate maintenance, packaging and distribution in good faith; and sometimes all it takes is just one package, for the whole system to become compromised. But even so, I’m more comfortable knowing, the majority of software I’m running on my machines, to be open-source; than relying on a single entity, like Microsoft, having an abysmal track record in respect of privacy, while doing so in the dark. Of course you could restrict access to Microsoft servers using network filtering, but it’s not just that aspect, it’s also not having to deal with Microsoft’s increasingly restricted experience, primarily serving their perverse dark patterns. I do believe people should handle sensitive files with care, for instance: put Tails on a live-USB, leave it off the internet, put the files on an encrypted drive, dismount the drives physically, and store them somewhere safe.

Ah sorry, it seems I read over that part. Unless programmers have the exceptional skills and time required, to effectively reverse engineer these complex algorithms, nobody will bother to do so; especially when required after each update. On the contrary, if source code was available, the bar of entry is significantly lower and requires way less specialized skills. So save to say, most programmers won’t even bother inspecting a binary, unless there’s absolutely no other way around or have time to burn. Where as, if you’d open up the source, there would be a lot more, let’s say C programmers, able to inspect the algorithm. Really, have a look at what it requires to write binary code, let alone reverse engineering complicated code, that somebody else wrote.

I agree with Linus’ statement though: I rarely inspect source-code myself, but I find it more comforting knowing, package-maintainers for instance, could theoretically check the source before distribution. I stand by my opinion that it’s a bad look for a privacy- and security-oriented piece of software, to restrict non-“experts” from inspecting that, which should ensure that.

If open-source, a lot more eyes could be on it, and therefore the chances of intentionally implemented vulnerabilities, by Threema itself, would have a higher chance of being noticed before able to be exploited, by both hackers and Threema (partners).

Still, if the service is supposed to be security and privacy-oriented, how about you make the source-code available, so users can verify this for themselves?

proprietary encryption algorithms verified by thought-leading cybersecurity experts and communities worldwide

Trust the experts bro

AI risk management, what could possibly go wrong?

Of course, make an anti-feature part of an integral part, which coincidentally also happens to handle personal files…

Great article! “education” … “risk assessments” … “early intervention” … Got to break their spirit while they’re young

It’s almost as if they’re seeking to replace these with technology. They’ve purposefully neglected social services and will continue to do so, to lower the bar for AI and grant themselves an excuse for the poor “substitute”. And this isn’t at all restricted to the UK, in The Netherlands we’re in the midst of it too: the same exact playbook. Modern surveillance cameras (like Axis’ for example) have NPU’s built in, or camera footage (even from legacy analog cameras, by use of encoders) is linked to either an onsite server, a cloud-service, or a combination of the two, facilitating the functionality. I hardly believe AI to be the limiting factor here, storage of footage is another story however. But I think they instead strategically place facial-recognition cameras, while the other cameras simply store abstractions from the footage. Of course if one of those cameras senses an event, which it recognizes might be of elevated relevance, it might store the raw footage. An example being: railways doing face-scanning for “depression detection”, instead of implementing ‘platform screen doors’ of course…