"[The Dhāraṇī Chapter is] another later addition probably when Dhāraṇī was extensively taken into the body of Buddhist literature just before its disappearance from the land of its birth. Dhāraṇī is a study by itself. In India where all kinds of what may be termed abnormalities in religious symbology are profusely thriving, Dhāraṇī has also attained a high degree of development..." - D T Suzuki. The lankavatara Sutra. p.223, n.1. [Image] D. T. Suzuki D T Suzuki is a key figure in the translation of Buddhism to the West. He is cited by David McMahan as representative of the Romantic influence in Modernist Buddhism. However this quotation and those below show that he was also under the influence of Scientific Rationalism. Though perhaps his response to dhāraṇī as an "abnormality in religious symbology" is rather more emotional than rational.

Suzuki is not alone in struggling to find a place for dhāraṇī in Buddhism. In his History of Buddhist Thought E. J. Thomas suggests that dhāraṇī has "infected popular Buddhism" and "spells form an important part of popular Buddhism, but they have nothing in themselves peculiarly characteristic of Buddhism. They are a form of sympathetic magic” (186. Emphasis added). So far from the Western conception of Mahāyāna Buddhism is the dhāraṇī, that it gets no mention in Paul Williams' oft cited textbook Mahāyāna Buddhism, despite being a central element in Mahāyana Buddhism as it is practiced around the world.

Suzuki is equally vehement when it comes to the dhāraṇī in the Heart Sutra which he refers to as "apparently a degradation or a degeneration" (228). [1] “A Mantram or Dhāraṇī is generally supposed, when uttered to effect wonders… Can we say, then, that the end of the Buddhist disciplines can be attained by means of a mere mystic phrase?” (Suzuki Essays. 229) "Another thing which makes this presence of a Mantram in the Hṛidaya more mystifying is that the concluding Mantram is always recited untranslated as if the very sound of the Sanskrit-Chinese were a miracle working agency." His polemic against the Heart Sutra dhāraṇī is quite lengthy and concludes: "taken in itself has no meaning, and its vital relation to the Prajñāpāramitā is unintelligible”. (Suzuki Essays. 236) The conceit that if I cannot understand a subject then it is "unintelligible" is common in scholars of Suzuki's generation and before. However, contemporary scholar, Donald Lopez, is likewise puzzled, and so he tells us are the Indian commentators whose works are preserved in Tibetan. He says: "The question still remains of the exact function of the mantra within the sūtra, because the sūtra provides no such explanation and the sādhanas make only perfunctory references to the mantra". (Lopez. The Heart Sutra Explained. p.120) The question about the role of the dhāraṇī in the Heart Sutra raises a deeper question. We know that from the second century CE a large number of dhāraṇī sūtras were composed and that they were clearly very important to Buddhists of the time and some of them continue to be important in Asian Mahāyāna Buddhism (A Zen practising friend recently sent me a dhāraṇī for removing obstacles to help me find a new place to live!). The collection of Nepalese manuscripts at Hamburg University lists hundreds of dhāraṇī sūtras! We know that from around that time or a little later, dhāraṇīs began to be added to Mahāyāna sūtras such as the Laṅkāvatāra, the Saddharmapuṇḍarīka and the Suvarṇaprabhāsottama. Given the apparent prominence of dhāraṇī in Mahāyāna Buddhism from relatively early in the development of it, why are modern commentators on Mahāyāna surprised to find a magical chant in a Buddhist sūtra? Let's try to deal with this question first, and then look at the specifics of the Heart Sutra.
Buddhism is inseparable from magic. Many of the most interesting stories in the life of the Buddha involve magic and miracles. The story of the conversion of the Kassapas at Gaya involves a whole series of magical feats for example. The very awakening experience is often retold in terms of the development of specific super-normal powers such as knowledge of past-lives or clairvoyance (and plenty of living Buddhists claim to have experienced these magical powers). As time goes on more and more miracles and magic are added to the stories, but there is no point at which magic and the supernatural is not evident. There is no absolutely no reason to assume that 'original' Buddhism was a purely rational affair. This is a Western conceit that has distorted Buddhism from at least the 19th century onwards. The idea of "rational Buddhism" is ironic because the initial European view of Buddhism was that it was a form of irrational, idolatrous, superstitious, heathen religion (this is brought out in Philip C. Almond's The British Discovery of Buddhism[Image]).
As Westerners we have developed an ambivalent attitude to magic since the European Enlightenment. There seems to be a spectrum of belief from entirely credulous to entirely sceptical, but on the whole we consider magic to be a form of entertainment, not to be taken too seriously. Or we treat it as a vulgar relative of the religious miracle.
Be that as it may there is no getting away from the facts: Buddhist texts are full of magic. And in particular they are filled with that magic which involves spoken sounds. I've recently made reference to the paritta and to the sacca-kiriya or 'truth act', both of which are encountered in early Buddhism. I'll be saying more about the sacca-kiriya in a couple of weeks.
The most obvious answer to the question of why anyone would be surprised to find a magic spell in a Buddhist text is that they are fooling themselves. Magic and superstition is exactly what we ought to expect. But more specifically the presence of a dhāraṇī in a late Mahāyāna text is exactly what we expect to find. Mahāyāna texts are full of dhāraṇī. 
From as early as the second century sūtras were being composed specifically as a vehicles for dhāraṇī. By the fourth century those texts which did not have one, had them added (which is what Suzuki is referring to in the first quote). A number of dhāraṇī continue to be popular in China and Japan today: e.g. the Nīlakaṇṭha Dhāraṇī or the the Karaṇḍamudrā dhāraṇī. Several times a year people write to me to ask if I have the Uṣṇīṣavijaya Dhāraṇī in Siddhaṃ script (I don't). If you look around there are dozens of YouTube videos with the dhāraṇī being chanted in Chinese transliteration.
The presence of dhāraṇī in Buddhist texts is used by some to argue that Tantric Buddhism dates from as early as the second century, but it really doesn't. Tantra is a synthesis of a huge variety of religious ideas and forms which happened in India in the 6th-7th centuries CE. Ronald M. Davidson plausibly argues that the synthesis was given impetus by the breakdown of socio-political structures across India with the fall of the Gupta Empire and the resulting chaos. This places the Tantric synthesis in the late 5th century at the earliest, but the 6th century is more likely. The first  fully developed Tantric text, Mahāvairocana Abhisambodhi Tantra, dates to the mid 7th century (probably the 640's) according to Stephen Hodge. The cult of dhāraṇī predates Tantra by several centuries.
The fact is that our information about Buddhism is curtailed (to be generous, or censored to be cynical). Thus dhāraṇī centred Buddhism does not feature much in Western narratives because it not only does not put aside superstition, but places magic at centre stage. I've argued at some length that most Buddhism is sharply dualistic, with belief in a distinction between matter and spirit, and a profound bias for spirit. Magic is largely, though not completely, excised from modernist Buddhism. It is particularly rigorously rooted out when it has any connection to the mundane or physical world - spirit magic is more acceptable. Using Buddhist (magical) techniques for mundane purposes is frowned on, even though this has been the norm for millennia. Many Western Buddhists are dismissive of Soka Gakkai for precisely this reason. Mindfulness therapies are also attacked because for commodifying the Dharma and their goals are oriented towards material well-being rather than transcendence of suffering (though some practitioners, with varying degrees of plausibility, deny these charges).
Something to keep in mind is that mature Mahāyāna held that awakening required three incalculable aeons to accomplish. There were times and places where this made awakening seem impossibly far off. When awakening is absolutely transcendental and thus of infinite value, it is also beyond the reach of ordinary mortals. Under these conditions intellectualism and scholasticism seems to flourish. Buddhism becomes a rather academic exercise for monks, and a way to ward off evil for lay people; and after all, lay people are not expected to be actively pursuing awakening in most Buddhist societies anyway. This was certainly true of, for example, the Mahāyāna Buddhism in late 8th century Japan. It stands out because it was against this backdrop of impossibly far off awakening, and absolutely separate Dharmakāya, that Kūkai and Saichō introduced Tantra to Japan. Kūkai's claim that tantra was a communication directly from the Dharmakāya and his maxim 'awakening in this very life' at first caused confusion and resistance in the Japanese Buddhist establishment, before setting off a religious revival that continues to inform Buddhism in that country.
Our pre-scientific forbears lived in dangerous world at the mercy of disease and elements. They knew as we know, if we are honest, that we are not in control. We don't even control our own bodies which age and get sick against our will. The attraction of any technique which claimed to exert control over nature and the cosmos, which held chaos and danger at bay, ought to be obvious. In our own time a man was recently jailed because he sold a device consisting of an aerial connected to a plastic handle to the Iraqi military as a bomb detector. The units sold for more than US$30,000 each, and the Iraqis are said to have bought about 6000 of them. Faced with uncontrollable forces we are desperate for some protection.
When my appendix grumbled and caused me intense pain, I did not resort to chanting the Heart Sutra, I high-tailed it to a hospital and had them remove the offending organ under a general anaesthetic before it killed me. But in the ancient world the Heart Sutra might have been all that I had. It would not have saved me, but it might have made me feel better for the short time before I died of septicaemia. It might have soothed my relatives as they watched me die in agony. I know that many moderns argue for the efficacy of ancient medicine. It's true that chewing willow bark would ease pain because it contains salicylic acid. But without any knowledge of the mechanisms of the body or of the chemistry of such remedies there was no way to be truly systematic about it. We see this in the medieval medical approaches such as humours, acupuncture, and āyurveda. All of these approaches to wellbeing are unrelated to the way the body actually works and rely on magical explanations to explain their efficacy. There is an irreducible element of magic because when they were invented magic was the most potent counteractive to the uncontrollable forces in those peoples lives. At least they believed it was. And in the face of violent or painful death who can blame them? It's harder to explain their persistence or the emergence of a non-system like homeopathy. 

How Should We Understand Magic?
What is magic? By which I mean what is it really? I don't take magic on it's own terms. Indeed I don't think we should take any form of tradition on its own terms. However, understanding the persistence of magic requires careful consideration. On the face of it magic simply does not do what it says. Hence it's value almost certainly lies elsewhere. Over the years a number of Western attempts to understand magic have suggested that believers were simply childish, idiots, delusional, or wishful thinkers; or that magic was meaningless symbolic action. But none of these theories are enough to explain the persistence of magic, particularly in the realm of healing. Most of them reflect Western intellectual laziness more than informed comment. My own preference is for the work of Ariel Glucklich. His in-aptly named book The End of Magic[Image], offers a critique of Western attempts to explain magic and, on the basis observation of modern Tantric healers in India, tries to do better. I touched on this back in 2008 in an essay called Mantra, Magic, and Interconnectedness. Glucklich summarises: "Magic is based on a unique type of consciousness: the awareness of the interrelatedness of all things in the world by means of simple but refined sense perception... magical actions... constitute a direct, ritual way of restoring the experience of relatedness in cases where that experience has been broken by disease, drought, war, or any number of other events." (12) The theme of interrelatedness is one that is deeply resonant for Buddhists. Even before the direct theme of interconnectedness emerges in Buddhism we find descriptions of the experience of radiating love, treating all beings as a mother treats her only child, and making no distinctions between self and other, etc. These are redolent of one of the most common of all so-called mystical experiences, oceanic boundary loss, in which one feels at one with everything, completely selfless, and blissful. I think this experience is central to understanding Indian religion and seems to underlie lot of the religious impulses, certainly in Buddhism. We reify and cheapen the experience with maxims like 'all is one', but the experience of being at one with everything ought not to be dismissed lightly. I suggest that if we are seeking a rational explanation for the purpose and persistence of magic in Buddhism this is where we might usefully start.
Another observation I've made about dhāraṇī is that the early dhāraṇī show every sign of not being in Sanskrit, but of being from a Prakrit (or vernacular) tradition. When we see a dhāraṇī like the one that Suzuki was complaining about in the Laṅkāvatāra... Tuṭṭe tuṭṭe vuṭṭe vuṭṭe paṭṭe paṭṭe kaṭṭe kaṭṭe amale amale vimale vimale nime nime hime hime vame vame kale kale kale kale aṭṭe maṭṭe vaṭṭe tuṭṭe jñeṭṭe spuṭṭe kaṭṭe kaṭṭe laṭṭe paṭṭe dime dime cale cale pace pace badhe bandhe añce mañce dutāre dutāre patāre patāre arkke arkke sarkke sarkke cakre cakre dime dime hime hime ṭu ṭu ṭu ṭu ḍu ḍu ḍu ḍu ru ru ru ru phu phu phu phu svāhā. ... our first thought is not Sanskrit literature or any other kind of high culture. The doubled retroflex consonants and -e endings point away from Sanskrit as the language of composition and towards a Prakrit like Māgadhī. This form is all about rhythm, repetition and alliteration, and lacks any formal grammar or semantics. The point of this is not what it means because this is not language as such. The point is that in chanting it one has an experience that will be partly determined by culture and conditioning and partly by circumstances. This dhāraṇī is meant to be chanted aloud, repeatedly, probably collectively in a ritual context. And the experience of chanting it, is the point of chanting it. We have very little information about the way these sounds were understood or used. 
These sounds were probably made to protect the chanter from misfortune, not to gain any rarefied spiritual attainment. This is about survival, about invoking chthonic forces to come to your aid in times of trouble. It is something to chant when feelings of being isolated or alienated from your people, family, or tribe threaten to overwhelm you. And we know from the literature of China that the Heart Sutra itself was used this way. If Glucklich is right then the chant protects the chanter by (re)establishing their sense of connection to everything.
We don't, or no longer, live in a culture where is makes any sense to approach interrelatedness in quite this way. However, we are not a million miles from it either. For example, we still sing together and through that experience, especially in large gatherings can experience the relatedness that music or chanting facilitates. Rhythmic chanting stimulates endorphin production, especially when done in groups, but the effect must have gone beyond a feel good factor. I've done exercises with groups which encourages a careful listening to the sounds of chanting, and stimulated receptivity to those sounds, and achieved a heightened sense of both calm and connectedness. It's possible to use vocal sound in this way because of the way that it affects us. Non-word vocal sounds both evoke and communicate emotions without involving the intellect (Compare the study of conversational grunts). I believe that there are good evolutionary explanations for this, but the experience itself is far more persuasive. It is enough for most people to chant together and experience a sense of connectedness to their fellows.
This experience is overlaid with various ideas in whatever culture is experienced in. Ancient Indians no doubt believed in the magical efficacy of chanting and with the development of Tantra layered meaning onto the spoken or chanted sound. In a reading culture we forget that the Dharma was primarily an aural experience for most of the last 2500 years. To be well versed in the Dhamma was to be a sutavant 'one who has heard' or a savaka 'a hearer'. The divine revelations of the Veda were called 'śruti' 'hearing'. One did not read silently in our culture until relatively recently, and even now we hear the words in our heads as we silently read them. We probably all know that a well spoken poem, for example, is a very different aesthetic experience to a silently read one.
The dhāraṇī cult left behind little or nothing about the mechanics of dhāraṇī - so far as I am aware. While the Arapacana alphabet and the idea of mnemonics are often invoked they are seldom relevant except in that specific context. Nor is the idea that the dhāraṇī represents a condensation of the text found in actual texts to my knowledge (except perhaps in the later dhāraṇī texts). Neither explanation obviously applies in the case of the Heart Sutra. By the time dhāraṇī start appearing in Buddhist texts it's almost as if their origins are already forgotten. One can see the forgetfulness in the development of the Arapacana as it is expressed in the Sanskrit language - the seeming ignorance that it is the Gāndhārī alphabet is already apparent in the Sanskrit Pañcaviṃśati. Indeed it's quite difficult to tell what the word dhāraṇī means from the way it is used in texts. The etymology is clear enough, it's from √dhṛ 'to bear' and thus cognate with dhara 'bearer' and dharma 'foundation'. However, for example when we try to understand it in context in a text like the Pañcaviṃśati the impression we get is quite vague. We only know that it is associated with these strings of syllables, but not how it is associated or why.


Conclusion
To sum up, the Heart Sutra has a dhāraṇī in it because it was de rigueur for a Buddhist text of that era to have one. To have composed a text and not included a dhāraṇī would have been odd by the standards of the time and place. Every Buddhist, particularly in China, would have had a favourite dhāraṇī or two that they chanted for protection and prosperity. Dhāraṇīs were a prominent feature in monastic liturgy.

The early dhāraṇī probably emerged from a Prakrit speaking milieu and to some extent they retained Prakrit features (such as the -e ending) but were gradually Sanskritised as Sanskrit became the standard language for Buddhist texts in India. Dhāraṇīs invoke a kind of magic which we no longer understand - partly because no records were made of how it worked, and partly because it was later overlaid by Tantra. However, Ariel Glucklich theory of magic gives us a way to understand dhāraṇī chanting with a degree of rationality that does not diminish the experience of the practice.

The Prajñāpāramitā texts the Heart Sutra was made from lack dhāraṇīs (pre-dating the cult by a century or two), so a suitable dhāraṇī was found in other popular texts and adapted for the purpose. Since it had no contenders, it became the de facto Prajñāpāramitā dhāraṇī. The relationship to Prajñāpāramitā does indeed seem to be arbitrary, though I suspect the initial choice had meaning for the monk who chose it.
~~oOo~~

Notes
I have tried to show that the so-called mantra in the Heart Sutra is in fact a dhāraṇī, and that the word 'mantra' itself a substitute for 'vidyā' and is used in the text only because of a confusion caused by being translated back and forth between Sanskrit and Chinese at different times.
For more reading on the role of dhāraṇī in Chinese Buddhism I recommend this article which turned up after I finished writing this essay:
Mcbride, Richard D. II (2005) 'Dhāraṇī And Spells In Medieval Sinitic Buddhism.' Journal of the International Association of Buddhist Studies. 28(1): 85-114. Online: http://archiv.ub.uni-heidelberg.de/ojs/index.php/jiabs/article/viewFile/8958/2851 Had I read this earlier I would have incorporated it into the previous essays.


Bibliography
Davidson, Ronald M. (2003) Indian Esoteric Buddhism: A Social History of the Tantric Movement. Columbia University Press. Glucklich, Ariel. (1997) The End of Magic[Image]. New York, Oxford University Press.  Hodge, S. (trans.) (2003) The Mahā-vairocana-abhisambodhi tantra : with Buddhaguhya’s commentary. London : Routledge Curzon.  Suzuki, D. T. (1971) Essays in Zen Buddhism: third series. Red Wheel/Weiser. [First published 1934] Suzuki, D. T. (1991) The lankavatara Sutra: a Mahayana text. Taipei : SMC Publishing. [First published 1932]