The test was supposed to validate the sensor. DARPA — the Pentagon’s advanced research agency — had built a system under its Squad-X programme to detect approaching humans, and eight US Marines were invited to try to beat it. They did not jam it, hack it, or spoof its electronics. Two somersaulted their way across the open ground. Two others advanced underneath a cardboard box. One walked towards the sensor dressed as a fir tree. Every one of the eight reached the objective undetected.
Paul Scharre, the former Army Ranger who helped write the Pentagon’s rules on autonomous weapons, tells the story in Four Battlegrounds — and pairs it with the finding it illustrates: “AI technology is powerful, yet its power is a brittle one.” The system had learned to recognize the humans it had been shown. Humans who walk. Nobody walked.
Two things about the episode deserve more attention than the laughter it usually earns. The first is the cost structure. The sensor represented years of funded research; the countermeasures were invented in an afternoon, for the price of a cardboard box. That asymmetry is not a punchline — it is the standing condition of any learning system placed in front of an adversary who is allowed to move.
The second is the discovery mechanism. No methodology produced the somersault. What produced it was people who wanted to win, probing a rule-bound system until it gave way. Armies own that resource in unlimited quantity, and mostly aim it at the enemy’s equipment. The rarer discipline is aiming it at your own — before fielding, not after.
The Marines did not out-compute the algorithm. They stepped outside its training data — and found the door was never locked.