JAAN TALLINN spent this past week in New York at the United Nations General Assembly, his latest stop in a year of long-haul trips to tell entrepreneurs, investors, policymakers and anyone else who will listen: AI could end humanity.
One of Anthropic's earliest investors, Tallinn stands to make billions in the coming months from the artificial-intelligence company's expected initial public offering. Concerns about AI motivated him to invest.
On the February trip from an AI summit in New Delhi to an ethical AI conference in Paris, Tallinn chartered a jet for a group that included AI researchers Stuart Russell and Yoshua Bengio and a classical pianist named Serene. They debated machine consciousness at 40,000 feet over flutes of champagne.
For over a decade Tallinn has been making the case for AI safety that spilled into public view in recent weeks when Anthropic researcher Jacob Coxon resigned and warned that AI could spin out of control.
"Unfortunately, I believe we are not taking this threat seriously enough and that we are racing to build a technology that could end humanity," Tallinn said in a statement. "If more people demand the right to understand the risk and speak up about the risks of recklessly developing this technology, then we and our children will all stand a better chance of surviving."
Tallinn has many fears: the race between AI labs, the U.S. rivalry with China, AI's growing ability to improve itself. He has imagined apocalyptic scenarios like AI data centers snapping up land once used for agriculture and causing mass-starvation or AI engineers altering the earth's atmosphere, making it inhospitable to humans.
Tallinn's view, according to friends and colleagues, is that if he didn't invest in Anthropic, an investor unconcerned with safety would have taken his place. He believes he can have the most positive impact by investing in AI companies, and using his returns to invest in nonprofits that support AI safety.
The AI-safety community is divided over whether the best way to encourage safe AI development is to work at or invest in the companies developing the technology, or to support independent safety research without fueling the AI race.
"All good, talented engineers who are mindful of that risk basically have a decision to make," said Rauno Miljand, the managing partner at Tallinn's investment firm, Metaplanet: "Run to the shelter and try to spend as much time with loved ones, or to try to impact the process and steer the probabilities that humanity will survive."
TALLINN, 54, was born in Estonia-the small Baltic country with a tech hub that punches above its weight-and raised behind the Iron Curtain, an upbringing that encouraged his belief in entrepreneurship, free expression and information sharing. His parents were an architect and a film director. A self-described super nerd, he first got access to a computer at age 14.
After earning a theoretical physics degree, he helped build the file-sharing application Kazaa, which introduced music piracy to many, followed by Skype, in the early 2000s, which Microsoft later bought for $8.5 billion.
Tallinn was searching for his next project in 2008 when he stumbled onto the work of AI researcher Eliezer Yudkowsky, a leader of the rationalist community, an intellectual movement known for being concerned about the risks of AI. Yudkowsky is famous for his prolific writing on AI alignment; his 660,000 word fan fiction, "Harry Potter and the Methods of Rationality;" and a recent book "If Anyone Builds It, Everyone Dies."
After debating with Yudkowsky, Tallinn was fully convinced of AI's catastrophic potential. Tallinn became a major donor to AI-safety organizations, including Yudkowsky's Machine Intelligence Research Institute, as well as effective altruist groups such as the Centre for Effective Altruism. Effective altruism is a philosophy focused on doing good in the world, including through charitable giving, and preventing AI from doing harm.
Ranked Estonia's wealthiest person by local media, with an estimated $12 billion fortune, he's funded AI safety initiatives with about $170 million, according to a list he published of his donations. He also co-founded the Cambridge Centre for the Study of Existential Risk and the Future of Life Institute.
In 2011, Tallinn connected with Holden Karnofsky, who co-founded the charity evaluator GiveWell and Dario Amodei, who was working with the organization, to discuss AI risks. During a phone call with the two future Anthropic leaders that same year, Tallinn made a bleak prediction he's become known for, according to a transcript published by Coefficient Giving, which grew out of GiveWell.
"I estimate the probability of us dying from an AGI disaster in double digits," he said, according to the transcript. "To me it's as though we are trying to defuse a bomb."
Amodei was less certain: "The way I think about it, there may or may not be a bomb," he responded. "Maybe the actions we associate with defusing the bomb are the ones most likely to set off the bomb."
Although they disagreed, Amodei and Tallinn stayed in touch.
Tallinn kept investing. He was an early backer of the AI lab DeepMind and served on its board before it was acquired by Google in 2014.
"In the case of DeepMind, the founders really didn't want to listen to him and didn't," said Sebastian Mallaby, author of the book on DeepMind, "The Infinity Machine." "He didn't succeed in having any influence."
That said, DeepMind's success earned Tallinn clout. Over time, the scope of Metaplanet's investments expanded and now include SpaceXAI, the Boring Company and smart ring company Oura, which is also planning an IPO.
He cemented a reputation among friends and fellow investors as someone who would voice his concerns often and openly, including with those who didn't share them.
People who know Tallinn describe him as surprisingly upbeat, despite his deep fears of an AI apocalypse. He takes solace in movies-the Richard Linklater romance "Before Sunset" is among his favorites-hip-hop dancing and learning Japanese. He trains regularly with a dance instructor and practices freestyle dancing, from San Francisco to his favorite city, Tokyo.
"He literally goes dancing, sleeps for four or five hours, comes in and gives a talk at [charitable organization] The Elders, goes home, takes a nap and then does some meetings over dinner," said Sean Ó hÉigeartaigh, a Cambridge University AI researcher and friend. "If this stuff wasn't going on, he would spend all of his time dancing and coding."
IN 2021, shortly after Amodei and others exited OpenAI in part over safety concerns and started a new company called Anthropic, Tallinn agreed to lead a $124 million Series A investment. The round also included Facebook co-founder Dustin Moskovitz, Jane Street's James McClave, the Center for Emerging Risk Research and former Google CEO Eric Schmidt. It valued the young startup at $461 million, according to data firm PitchBook.
Absent from the group: traditional venture capitalists. Much of the venture capital establishment didn't put money into Anthropic because they viewed it as a David to OpenAI's Goliath.
Tallinn backed Anthropic in hopes his perspective would be heard in rooms where dangerous technology was being developed, he has said in interviews. Though influencing Anthropic's leaders also proved difficult.
As it grew larger, individual stakeholders had "less and less voice," Tallinn said in an episode of the Manifold podcast this year.
Miljand, the Metaplanet managing partner, has been trying to figure out how to use the proceeds the firm anticipates from an Anthropic IPO that could top $2 trillion in support of humanity, he said.
And if Anthropic doesn't succeed after all, Miljand said he and Tallinn will still have reason to celebrate.
"If it turns out that Anthropic fails and goes to zero, then that's a good thing because then we probably didn't reach ASI and humanity just bought itself additional years," said Miljand, referring to artificial superintelligence, or AI more capable than humans.
"It's kind of a win-win," he said.
Comments