The end of the Cold War, the growth of international terrorism and the arrival and subsequent explosive growth of the internet radically changed the environment in which GCHQ, UK's Signals Intelligence (Signit) agency, had to operate. Recognition of these pressures led GCHQ to begin a wide-ranging change programme in the late 1990s.
It included not only technology but also deep change in business processes, leadership and culture. A critically important feature of these changes has been the adoption, and when necessary adaptation, of a wide range of management techniques taken from the private sector.
The intelligence world, by its very nature, is shrouded in secrecy. Little is said in public about the management of intelligence agencies and of the processes they use.
However, their business is complex and requires careful management if they are to be effective, properly accountable and secure.
Signals intelligence in particular demands sophisticated managerial techniques, because it involves advanced technology, large numbers of people, enormous volumes of data, and large amounts of intelligence output – a combination that cannot be managed casually.
Policy administration has undergone a radical transformation in response to fundamental changes in the geopolitical and technological environment in which it operates.
As a result, it is now well equipped for the challenges of the 21st century. One important feature of the transformation has been the adoption of a wide range of managerial practices and techniques from the private sector – a marked change from its previous approach.
Before the transformation process began, the GCHQ culture was very much a product of the Cold War, and of the Civil Service culture of that period. It was a static organisation, essentially because its main target, the Soviet Union, was static; in 1990 the basic structure had not changed for 30 years, and organisational change was regarded as a major venture.
The organisation was also very inward looking. It made little or no use of external management techniques, but relied upon a long tradition of home-grown methods. This too became a real disadvantage as the pace of change increased.
Drivers of change
At the start of the 1990s, the Cold War was ending and GCHQ's world had begun to change. Not only did the all-embracing intelligence requirements on the Soviet Union disappear, but even requirements for classic military intelligence on the new Russia diminished rapidly.
So large parts of the organisation, and much of the knowledge in people's heads, became redundant.
In place of the USSR, GCHQ faced increasingly volatile requirements on a host of new subjects – such as the proliferation of weapons, both conventional and weapons of mass destruction (WMD), the narcotics trade, people smuggling, and eventually international terrorism on a new scale.
There were demands to support military operations in new places, such as the Balkans. These changes posed a major challenge for an organisation that had no experience of radical change, and for managers who had learned their trade in a relatively slowly-changing environment.
Communications structures before the internet can be thought of as fixed networks in which any 'message' (telephone call, fax, or data transmission) travelled over a single identifiable path between two communicators. Internet technology, on the other hand, relies upon a complex and frequently changing global network in which there are many routes between any two points; and it breaks every 'message' (a term which now includes emails and web-browsing sessions) down into a series of small packets, which travel independently over the network, with different packets able to take different routes, before being reassembled by the intended recipient.
This change requires an entirely different technical approach by interceptors, who can no longer rely upon having fixed paths for the messages they seek. It was immediately apparent in the late 1990s that GCHQ would have to embark upon a major redesign of its technology.
But technology was only part of the answer. Careful examination of the way the internet operates, and of the way its users behave, brought about the realisation that what was needed was not just a new technical infrastructure, or even new organogram, but a radically different type of organization.
GCHQ had to be able to cope with the fact that virtually all its targets were now using the same technology, and that techniques developed against one target were likely to be of direct relevance to many others. This insight led to the initiation in late 1997 of a wide-ranging change programme, involving many aspects of culture and business process.
The need was in due course multiplied by the dramatically increasing pace of technology change that the open nature of the internet facilitated, and again by the mobile telephony revolution.
The changes that GCHQ had already achieved, enabled it to respond to these new demands; but this response involved yet further radical rethinking of its business processes – in order, for example, to accelerate its own pace of technology development.
Over the early years of the new millennium there were yet more events and environmental factors that demanded a response: 9/11 and the London bombings, the Iraq and Afghan wars. All these served to produce further destabilisation in GCHQ's environment, and forced more (and more radical) rethinking of the way the agency operated.
The changes needed
The full story of the transformation would require a much longer article. It involved not just a huge technology programme, but also structured work over several years to develop new business processes and a new culture, a new building to enable those changes, and the governance structure and leadership skills to drive them.
It will be helpful to an understanding of the context for the main subject of this article to list some of the internal changes that were identified as necessary. This is a heavily summarized list of my top 10:
• The ability to manage large-scale change – something in which GCHQ had no real track record
• Greater agility and flexibility, to cope with a volatile world instead of a static one
• More knowledge sharing – because most targets were now using the same technology
• Better resource use – because the demands were growing faster than any likely growth in the budget
• Better planning and better delivery – for the same reason
• Better management of the IT infrastructure – because the systems were getting too complex for the current control methods to cope
• Faster and more efficient development of new technology, to keep pace with the pace of change in the IT world
• The ability to manage complex programmes that combined technology and business change
• New approaches to customer service – because in the wake of 9/11 and the Iraqi WMD story, Sigint customers began to expect and need a different relationship with intelligence producers
• Better leadership – to lead the organisation in a world in which dynamic change had become the norm.
Achieving these changes required the use of a wide range of techniques. Key elements of the transformation process have required the adoption of commercial best practice, ranging from corporate governance and HR to high tech-nology.
In no case did we find that this adoption was impossible, although on occasion we decided that benefits would not justify the effort required. In some cases minor adaptations have been required, and in a few cases the adaptations have been deeper, but the process of working them out invariably illuminated the underlying management issues.
GCHQ's non-execs and other external advisers have played a crucial role in the learning process.
Looking back, it is apparent that once the mindset changed, the scope for this approach became enormous. The first reaction to a serious management problem is no longer 'we have to invent a solution' but rather 'someone somewhere must have solved this already' – and our experience has been that these adopted solutions have done more for us than those we invented for ourselves.
Sir David Pepper Formerly the director of GCHQ
• This is an edited extract from: The Business of Sigint, The Role of Modern Management in the Transformation of GCHQ