Not far from the administrative offices in the Guantánamo Bay detention center, the CIA built eight small buildings that became known as “The Marriott” because of the comforts they offered in comparison to the conditions and spartan regime imposed on the prisoners in the main cells. With hundreds of prisoners arriving at the U.S. military base in Cuba to spend their days languishing in their cells, the CIA designed a program to recruit some of the most dangerous al-Qaida terrorists. The program was named Penny Lane, after the famous song by the Beatles.

Active and retired staff members from Guantánamo told the Associated Press news agency that the plan, which was implemented between 2003 and 2006, was to infiltrate these prisoners—on CIA salaries—into terrorist cells that they had previously been members of and convert them into U.S. government informants.

The plan was not risk-free. In fact, the risks were high: There was always the possibility that these men, once set free in their own countries, would betray the agency and conspire against the U.S. The program was created during the George W. Bush administration; there was also a danger that the former prisoners would become triple agents, or simply end up passing false information to the CIA in Langley, Virginia, that would lead to erroneous drone attacks on civilians in Yemen and Afghanistan.

There are no exact figures for the total number of prisoners who were recruited to Penny Lane, but it is thought that only around a dozen were considered and only a small fraction of these actually ended up working for the CIA. At its peak capacity, Guantánamo was home to 779 prisoners; today there are only 166.

The eight small buildings were built behind a little hill, further hidden from view by bushes and high cactus. Penny Lane was the counterpart to another secret installation in Guantánamo known as Strawberry Fields—after another song by the legendary British band—where “high value detainees” were housed, those who were never likely to be released and would be imprisoned on the island permanently, as suggested by the song’s full title, “Strawberry Fields Forever.”

Those who were part of the program had a life that could be considered privileged in the gulag that is Guantánamo, having access to such simple luxuries as real beds—the regular prisoners sleep on cots. The bungalows had private kitchens, showers, their own televisions and a small yard. According to AP, some prisoners requested—and received—pornographic material.

Although only a few prisoners were recruited as double agents, the program attracted the attention of President Bush, who interviewed a young man who had recently returned from Afghanistan, where the CIA generally liaises with its agents. Barack Obama was also aware of the CIA program, although in his case it was after he took power in 2009 and ordered an investigation of the ex-prisoner double agents and an analysis of its results, that he started to use Predator drones, according to a high-level AP reporter.

The CIA has put huge efforts and resources into infiltrating al-Qaida. Those who were candidates to swell the ranks of Penny Lane—the buildings can still be seen on satellite images—had to have genuine links with terrorism and, in order to be of use to the CIA, had to be able to re-establish these links with the group then led by Osama bin Laden.

The U.S. government had such high expectations for Penny Lane that an ex-intelligence agent remembers discussions about possibly releasing two Pakistani prisoners on U.S. student or business visas, in the hope that they would be able to set up links with al-Qaida and lead the authorities to members of a cell in the U.S. However, another high level ex-intelligence agent has denied this, according to the AP report.

There were various reasons why the prisoners agreed to work for the CIA, including the promise of safety and economic security for their families—the agency paid these agents millions of dollars through an account known as Pledge. Despite this costly investment, it is not known whether the double agents ever provided any useful information to the U.S. espionage network, or whether any al-Qaida members were captured or killed as a result of their work. Over time, some agents simply stopped passing information and disappeared off Langley’s radar.

The agents who were interviewed acknowledged that al-Qaida was always aware that this program might exist; as a result, its leaders tended not to trust those who returned from Guantánamo, which was a major threat to the success of Penny Lane.