The subhead of this Business Insider piece says it all:
Carnegie Mellon staffed a fake company with AI agents. It was a total disaster.
Some more gems:
It's relatively easy to teach them to be nice conversational partners; it's harder to teach them to do everything a human employee can.
And in conclusion:
Instead of being replaced by robots, we're all slowly turning into cyborgs.
As always in (tech) life: All that glitters is not gold. (for now at least)
Maybe not so good…
OpenAI's latest reasoning models, o3 and o4-mini, hallucinate more frequently than the company's previous AI systems, according to both internal testing and third-party research. On OpenAI's PersonQA benchmark, o3 hallucinated 33% of the time -- double the rate of older models o1 (16%) and o3-mini (14.8%). The o4-mini performed even worse, hallucinating 48% of the time. Nonprofit AI lab Transluce discovered o3 fabricating processes it claimed to use, including running code on a 2021 MacBook Pro "outside of ChatGPT." Stanford adjunct professor Kian Katanforoosh noted his team found o3 frequently generates broken website links.
Despite the seemingly negative headline, this is an incredible feat of engineering.
More than 20 two-legged robots competed in the world’s first humanoid half-marathon in China on Saturday […] The first robot across the finish line, Tiangong Ultra – created by the Beijing Humanoid Robot Innovation Center – finished the route in two hours and 40 minutes.
That last statement is much more important—not the attention-grabbing headline that the robots didn’t win against their human opponents:
Alan Fern, professor of computer sciences, AI and robotics at Oregon State University, told CNN he was 'actually very impressed' that the robots managed within the time limit, saying he 'would have bet that none of them would finish.'
Doxxing—The malicious practice of researching and publishing someone's private information online without their consent.
OpenAI’s latest release of ChatGPT has just made doxxing trivially easy: Upload a photo, ask ChatGPT to geolocate it, and chances are you will get a very precise location.
Want to know where your ex is partying this weekend? Just screenshot her Instagram Reels and let ChatGPT do the digging. To say this is messed up is an understatement…
There appear to be few safeguards in place to prevent this sort of “reverse location lookup” in ChatGPT, and OpenAI, the company behind ChatGPT, doesn’t address the issue in its safety report for o3 and o4-mini.
What happens when you create a rather powerful robot dog (the Unitree Go1), which is being used in all kinds of real-world applications – from surveillance and security to disaster recovery and beyond – and put a backdoor for easy corporate access?
Unitree did pre-install a tunnel without notifying its customers. Anybody with access to the API key can freely access all robot dogs on the tunnel network, remotely control them, use the vision cameras to see through their eyes or even hop on the RPI via ssh.
Not concerning at all…
These robot dogs are marketed at a wide spectrum of use-cases, from research in Universities, search and rescue missions from the police to military use cases in active war. Imagining a robot dog in this sensitive areas with an active tunnel to the manufacturer who can remotely control the device at will is concerning.
AI hallucinations can be hilarious in the best of cases, mislead you in others, and now create very real security risks when used in coding assistance (or even better, “vibe coding”).
Welcome to the age of slopsquatting: “[…] It refers to the practice of registering a non-existent package name hallucinated by an LLM, in hopes that someone, guided by an AI assistant, will copy-paste and install it without realizing it’s fake.”
And the problem is pretty darn real: “19.7% of all recommended packages didn’t exist. Open source models hallucinated far more frequently—21.7% on average—compared to commercial models at 5.2%. […] Package confusion attacks, like typosquatting, dependency confusion, and now slopsquatting, continue to be one of the most effective ways to compromise open source ecosystems.”
Better know what you are doing when you code your next app.
In case you haven't seen it yet (and are looking for some weekend reading), Stanford's annual AI Index Report for 2025 just came out.
The report is chockfull of insights about where artificial intelligence is heading.
Perfect timing for those of you wanting to dive deeper into understanding the current state of AI technology. I highly recommend setting aside some time this weekend to explore the findings.
Anthropic, the maker of the Claude foundational AI model, just released their fairly in-depth report on the use of their LLM by university students. Outside of the expected ("Students primarily use AI systems for creating (using information to learn something new) and analyzing (taking apart the known and identifying relationships), such as creating coding projects or analyzing law concepts”), the report admits that:
There are legitimate worries that AI systems may provide a crutch for students, stifling the development of foundational skills needed to support higher-order thinking. An inverted pyramid, after all, can topple over.
and
As students delegate higher-order cognitive tasks to AI systems, fundamental questions arise: How do we ensure students still develop foundational cognitive and meta-cognitive skills? How do we redefine assessment and cheating policies in an AI-enabled world?
These are very legitimate concerns – especially in a world that requires humans to be ever more on their A-game to keep competing with the very tool they use to outsource their learning.
Maybe our best bet against the AI apocalypse is to create better humans… With all the attention focused on AI, synthetic biology and genetic engineering are getting much less love – alas, the field is moving fast:
Working in the field of genetics is a bizarre experience. No one seems to be interested in the most interesting applications of their research. […] We've spent the better part of the last two decades unravelling exactly how the human genome works and which specific letter changes in our DNA affect things like diabetes risk or college graduation rates. Our knowledge has advanced to the point where, if we had a safe and reliable means of modifying genes in embryos, we could literally create superbabies.
Here’s what that means in practical terms:
Already with just 300 edits and a million genomes with matching IQ scores, we could make someone with a higher predisposition towards genius than anyone that has ever lived. […] Diabetes, inflammatory bowel disease, psoriasis, Alzheimer's, and multiple sclerosis can be virtually eliminated with less than a dozen changes to the genome.
And yet, we might be running out of time…
Given the current rate of improvement of AI, I would give a greater than 50% chance of AI having taken over the world before the first generation of superbabies grows up.
Many people, myself included, didn't try to build a product around a language model because during the time you would work on a business-specific dataset, a larger generalist model will be released that will be as good for your business tasks as your smaller specialized model.
This being said – with generalized models advances slowing down (see our post from earlier today), now might be the time to build:
If your business idea isn't in these domains, now is the time to start building your business-specific dataset. The potential increase in generalist models' skills will no longer be a threat.
Dean Valentine, co-founder of Zeropath, on using LLMs to conduct security penetration testing (which might serve as a good test case for LLMs' ability to “generalize outside of the narrow software engineering domain”):
Since 3.5-sonnet, we have been monitoring AI model announcements, and trying pretty much every major new release that claims some sort of improvement. Unexpectedly by me, aside from a minor bump with 3.6 and an even smaller bump with 3.7, literally none of the new models we've tried have made a significant difference on either our internal benchmarks or in our developers' ability to find new bugs. This includes the new test-time OpenAI models.
And on the reasons why AI models (and their makers) are falling short of their advertised claims:
AI lab founders believe they are in a civilizational competition for control of the entire future lightcone, and will be made Dictator of the Universe if they succeed. Accusing these founders of engaging in fraud to further these purposes is quite reasonable. Even if you are starting with an unusually high opinion of tech moguls, you should not expect them to be honest sources on the performance of their own models in this race. There are very powerful short term incentives to exaggerate capabilities or selectively disclose favorable capabilities results, if you can get away with it.
This doesn’t come as a surprise, nor is it entirely new - but seeing it in such stark light makes you wonder…
OpenAI (and likely everyone else) has a serious copyright problem. It makes me wonder when the copyright holders will start to actually fight back:
It’s a reminder that LLMs of this type and size all train on copywritten material.
Click through to the article from Otakar G. Hubschmann to see some of his findings – if you are not convinced that AI is trained on oodles of copyrighted material, this will make you see it.
Bill Gates said at a Goldman Sachs-sponsored event in L.A. two years ago:
"Whoever wins the personal agent, that will be a big thing because you’ll never go to a search site again. You’ll never go to a productivity tool again. You’ll never go to Amazon again. Everything will be mediated through your agent."
The first part is becoming true now – Adobe reported that traffic from generative AI sources (e.g., the Perplexity AI search tool) to e-commerce websites jumped a whopping 1,200 percent in their latest report.
In February 2025, traffic from generative AI sources increased by 1,200 percent compared to July 2024.
No wonder Google is simultaneously freaking out and investing heavily in an AI-powered future of search. But the important bit is not just the sheer increase in volume referred to by generative AI tools, but also the quality of this traffic:
[…] consumers coming from generative AI sources show 8 percent higher engagement as they linger on the site for a longer period of time. These visitors also browse 12 percent more pages per visit, with a 23 percent lower bounce rate. This speaks to the value of conversational interfaces in online shopping, which seem to help consumers be more informed and confident in their purchases.
While one party seems to never end, with OpenAI raising an eye-watering $40 billion at an even more eye-watering $300 billion valuation, other parties are being called off…
Microsoft is calling off some of its data center deals:
The software company has recently halted talks for, or delayed development of, sites in Indonesia, the UK, Australia, Illinois, North Dakota and Wisconsin, according to people familiar with the situation.
Microsoft is widely seen as a leader in commercializing AI services, largely thanks to its close partnership with OpenAI. Investors closely track Microsoft’s spending plans to get a sense of long-term customer demand for cloud and AI services.
One possible reason being discussed by analysts is the increasing efficiency that some AI models exhibit:
Analysts have stepped up their scrutiny of data center spending since Chinese upstart DeepSeek announced in January that it had created a competitive AI service using fewer resources than leading US companies.
Ostensibly written about Nvidia’s plight of having to move to bigger and bigger silicon as Moore’s Law ("the number of transistors per square inch on integrated circuits doubles approximately every two years”) is dead – this is, of course, much bigger than a single chip manufacturer.
Advancements in process technology have slowed to a crawl in recent years. While there are still knobs to turn, they're getting exponentially harder to budge. Faced with these limitations, Nvidia's strategy is simple: scaling up the amount of silicon in each compute node as far as they can.
[…]
In any case, Nvidia's path forward is clear: its compute platforms are only going to get bigger, denser, hotter and more power hungry from here on out. As a calorie deprived Huang put it during his press Q&A last week, the practical limit for a rack is however much power you can feed it.
Quoting Ned Batchelder from his blog post "Horseless intelligence”:
My advice about using AI is simple: use AI as an assistant, not an expert, and use it judiciously. Some people will object, “but AI can be wrong!” Yes, and so can the internet in general, but no one now recommends avoiding online resources because they can be wrong.
Ever wonder how AI actually "thinks"? A comprehensive research paper has explored the internal computational mechanisms of Claude 3.5 Haiku through advanced interpretability techniques. The findings reveal complex and surprising insights into how large language models actually perform computations.
The research shows that language models use sophisticated, parallel computational mechanisms. These often involve multiple reasoning pathways operating simultaneously.
Models exhibit remarkable abstraction capabilities. They develop features and circuits that generalize across different domains and contexts.
Internal reasoning processes can be quite sophisticated. They involve planning, working backwards from goals, and creating modular computational strategies.
Progress in AI is birthing a new kind of intelligence, reminiscent of our own in some ways but entirely alien in others. Understanding the nature of this intelligence is a profound scientific challenge, which has the potential to reshape our conception of what it means to think. The stakes of this scientific endeavor are high; as AI models exert increasing influence on how we live and work, we must understand them well enough to ensure their impact is positive.
[…]
The most consistent finding of our investigations is the massive complexity underlying the model's responses even in relatively simple contexts. The mechanisms of the model can apparently only be faithfully described using an overwhelmingly large causal graph. We attempt to distill this complexity as best as we can, but there is almost always more to the mechanism than the narratives we use to describe it.
A brave new world indeed…
Assuming this is not just a thing DoorDash did for marketing purposes, it surely points to a worrying trend:
DoorDash users can use Klarna to pay in four, interest-free payments or defer payments and let people pick a 'date that aligns with their paycheck schedules.'
AI (specifically large language models) need oodles of data to be trained on. AI companies thus gobble up as much data as possible – from publicly available data on the open Internet to copyrighted material from sources such as LibGen (a vast library of pirated books and scientific articles).
A longstanding practice for website owners to prevent bots from scraping their websites is the use of a robots.txt file – a small text file that you place in the root of your web host instructing (originally search engine) bots to only index certain parts of your website or stay away completely. Search engine operators such as Google or Microsoft honored these instructions – until the AI hype took hold and all rules went out the window.
Now AI bots aggressively crawl the Internet, ignoring instructions in robots.txt and even disguising themselves as regular Internet traffic to avoid being blocked – and it leads to problems beyond “just" stealing data:
According to a comprehensive recent report from LibreNews, some open source projects now see as much as 97 percent of their traffic originating from AI companies’ bots. […] Schubert observed that AI crawlers 'don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not.’ [Source]
In retaliation, Cloudflare, one of the largest network providers, lets its users fight back by deploying drastic measures:
"When we detect unauthorized crawling, rather than blocking the request, we will link to a series of AI-generated pages that are convincing enough to entice a crawler to traverse them," writes Cloudflare. "But while real looking, this content is not actually the content of the site we are protecting, so the crawler wastes time and resources.” [Source]
When it comes to AI, the gloves are most certainly off now…
The AI revolution isn't just about having tools - it's about knowing how to use them effectively. This simple 3-step hack transforms ChatGPT from a generic assistant into your personal subject matter expert.
Want ChatGPT to become an instant expert? Try this 3-step hack from Reddit.
First, ask for "20 words describing a [specific specialist]." This creates the foundation for your expert persona.
Next, request a 4-sentence prompt using those words to "summon this specialist." This crafts the perfect instruction set.
Finally, paste that prompt into a new chat and watch the transformation happen.
The result? Instead of generic bullet points, you'll get a conversational expert who guides you through complex topics with natural paragraphs and deeper insights.
Benn Stancil’s commentary about the changing nature of software development (vibe coding is all the rage now, kids!) draws an important conclusion: If developing software (aka writing code) becomes more and more democratized, what stronghold do places like Silicon Valley have on innovation?
Taking this thought a step further – if the value is less and less in the software development process and rather in domain expertise in the problem space, will we see a geographic shift of innovation ecosystems toward their respective client spaces?
Just as it's becoming harder to out-write an LLM, it's becoming harder to out-develop one too. And if experts can prompt their way to a product just as easily as those of us in Silicon Valley can, what winning talent are we left with?
Despite this blog post by Will Larson being written from the perspective of, and for, software developers, his insights into the impact of AI on careers (both from the perspective of an individual as well as a company) ring true across the spectrum:
The technology transition to Foundational models / LLMs as a core product and development tool is causing many senior leaders’ hard-earned playbooks to be invalidated. Many companies that were stable, durable market leaders are now in tenuous positions because foundational models threaten to erode their advantage. Whether or not their advantage is truly eroded is uncertain, but it is clear that usefully adopting foundational models into a product requires more than simply shoving an OpenAI/Anthropic API call in somewhere.
In our sessions, we often open with the observation that “we are trying to solve new world problems with old world thinking.” In Will’s words, our playbooks become rapidly obsolete, and in many cases, we haven’t developed new ones quite yet.
Sitting out this transition, when we are relearning how to develop software, feels like a high risk proposition. Your well-honed skills in team development are already devalued today relative to three years ago, and now your other skills are at risk of being devalued as well.
And as this world is moving at a frenzied pace, the above seems to be doubly true. As someone else recently wrote: Now might be the worst time to take a sabbatical.
A damning study from Columbia University, analyzing AI search engines' accuracy and their ability to cite sources:
Collectively, they provided incorrect answers to more than 60 percent of queries.
“More than 60% of queries” is pretty abysmal. It gets worse:
Most of the tools we tested presented inaccurate answers with alarming confidence, rarely using qualifying phrases such as 'it appears,' 'it's possible,' 'might,' etc., or acknowledging knowledge gaps.
On top of this, AI search engines also clearly have indexed material they were not supposed to (or more precisely, allowed to) access.
Perplexity Pro was the worst offender in this regard, correctly identifying nearly a third of the ninety excerpts from articles it should not have had access to.
It’s bad. Here is the study.
Fascinating conversation with Young Cho, head of research at the financial analysis firm Jane Street, on the challenges of using machine learning and LLMs in the context of financial data.
Machine learning in a financial context is just really, really hard... you can think of machine learning in finance is similar to building an LLM or text modeling except that instead of having, let's say one unit of data, you have 100 units of data. That sounds great. However, you have one unit of useful data and 99 units of garbage and you do not know what the useful data is and you do not know what the garbage or noise is.
Quoting Andrew Ng (who knows a thing or two about coding, AI and the future):
Some people today are discouraging others from learning programming on the grounds AI will automate it. This advice will be seen as some of the worst career advice ever given. I disagree with the Turing Award and Nobel prize winner who wrote, “It is far more likely that the programming occupation will become extinct [...] than that it will become all-powerful. More and more, computers will program themselves.” Statements discouraging people from learning to code are harmful!
In the 1960s, when programming moved from punchcards (where a programmer had to laboriously make holes in physical cards to write code character by character) to keyboards with terminals, programming became easier. And that made it a better time than before to begin programming. Yet it was in this era that Nobel laureate Herb Simon wrote the words quoted in the first paragraph. Today’s arguments not to learn to code continue to echo his comment.
As coding becomes easier, more people should code, not fewer!