The underlying intuition to first condition this question is that what is picked out by the word “philosophy” is basically the same as what is picked out by “theory.” “Theory” is whatever part of a discipline that reflects on the tools that discipline deploys to do its work. Thus, for example, “theory” is what is being done when a sociologist reflects on what is picked out by “society” or “group” or an anthropologist on “culture” or “ritual.” Almost every discipline these days has its own theory department, and no longer farms out the work to philosophy. (This didn’t used to be the case.) This has been a problem for philosophy, insofar as it has been harder for it to justify itself, but just insofar as it’s a problem for philosophy, it is—often covertly to those independent disciplines asking philosophy to justify itself—also a problem for the disciplinary theory-heads. The reason for this is that the experience of philosophy for 2500 years has been with abstract concepts and their relationships to each other. It has dabbled in concrete stuff from time to time, but every time it gets on a roll, the group of people in charge of the inquiry become full of themselves and secede from the union (a typical origins-story about disciplines from philosophy’s point of view). Without a doubt, every discipline has its own special problems that it needs its own special tools for, and for the most part it is best that a discipline make and improve its own tools (though outright stealing works, too). However, when it comes to seeing the relationship between those special tools and the special tools that other people are using, no one has any special province. The closest you can get is philosophy—that group of people with 2500 years of experience dealing with that kind of thing.
If you start with this understanding of “philosophy” and “theory,” the relationship between literary criticism and literary theory becomes a little clearer. It’s really the relationship between practical criticism of literary texts and theoretical reflection on the tools you use in those practical contexts. When you read Hawthorne through the lens of Freudian psychoanalysis or Melville through the lens of Derridean deconstruction, you’re doing practical criticism when the primary goal of your train of thought is reading Hawthorne or Melville. If you’re trying to make a comment on psychoanalytic theory or deconstruction using whatever insight you may have pulled from reading Hawthorne or Melville, the comment itself is theory and besides the point of reading Hawthorne or Melville. Getting the theory right is a different context than using it to read other texts. It’s important to understand this. In the act of writing, one can attempt to do many things. But to know you’re doing many things, or to just do one thing and not those other things, one needs to see how to differentiate between different activities. And in the terms I’ve laid out, “getting theory right” means doing philosophy, which means attending to the problems generated in a different disciplinary sector than that of getting the literary text right.
Literary theory as a phenomenon in English departments in the United States is the product of the confluence of five things:
1) concern over how interpretation worksThe last two are sociological, external to literary-critical practice as such (which is not to discount the impact of the external, e.g. coming from wealth, going to an Ivy League school, being white, etc.). The last in particular I shall pass over without comment, and about Continental philosophy I will say one thing: while Continental philosophy’s seemingly sudden vogue requires more extensive treatment on just why it and not something else (one prima facie reason is that it was more “humanistic,” in a non-technical sense, than analytic philosophy had become), the conjunction of (5) with the lack of an established conversation about post-Hegelian philosophy in the United States produced the need to establish one. So professors in English departments started bootstrapping themselves in. The trouble that we are now seeing, however, is that no one is teaching it. Someone needs to teach it, because we should be reading it, and if it has to be English departments, while being weird and ironic (say it with me: English departments), then so be it—but they need to do a better job at teaching it in-house. Right now it is scattershot at best, and even then hit-or-miss as to quality, but the pressure of the discipline is that you do need to know something about it. I think the pressure is abating, which is a good thing for the discipline in the long run, but it isn’t clear to me that we shouldn’t yet have a better quality baseline education in it.[fn.1]
2) concern over what interpretations we should give
4) the dearth of Continental philosophy in the American university system
5) Baby Boomers flooding the market and looking for a way of distinguishing themselves for tenure and seeing this new thing to do (thus flooding journals and presses with articles and books about it)
(1) through (3) are all theoretical, but it has been the conflation of these three distinct things that has produced the most theoretical trouble for literary theorists over the last 50 years. (3) is a distinctively philosophical issue, produced by thinking about “how one knows” isolated from any actual practices of knowing (it’s the isolation that makes it distinctively philosophical). The last 100 years of philosophy has produced a pure antifoundationalist strain of philosophical thinking. Philosophers of an antifoundationalist bent (which we can find all the way back in the Sophists and up to Hume and Hegel) finally reached self-consciousness about foundationalism being an actual problem to focus on in Nietzsche and the American pragmatists, and since then have produced a purer and purer polemic against that particular problem. The most confusing thing of all about the history of literary theory in the United States is that “literary theory” became self-conscious as a subgenre at the same time as the vogue of Continental antifoundationalism—which at the time was comfortably synonymous in many quarters with anti-theory (because “theory,” consciously or unconsciously, was nearly synonymous with “offering a foundation”).
The reason this could happen is that the dominant theoretical paradigm in English departments in the first half of the 20th century was “New Criticism.” New Criticism prided itself, however, on not being theoretical. The New Criticism achieved dominance in English departments, roughly, in part because of the pressure to discipline the discipline of literary studies (which itself was still being born as a university department in the first place), and so lose all the extraneous concerns in being a generally cultured person. What New Criticism combined, we can now see in hindsight, was the practice of “close reading” with foundationalism. To create a distinctive discipline, they claimed plausibly that the practice of close reading was what they did, implausibly that this was the only way one should read, and wrongly that this was the only way one could read. The first two claims are disciplinary claims for literary critics, and so can properly coalesce into a distinctive kind of thing called “literary theory.” The plausibility of their claims is, of course, a judgment by a disciplinary member on the far side of New Criticism’s hegemony. The third claim—the one called “wrong”—is the philosophical claim of foundationalism, where the moral “should” of the second claim turns into the stronger “could” of metaphysical necessity. (I’m not going to articulate and argue for why the vocabulary of correctness is, or rather should be, in play for the philosophical question as opposed to the disciplinary questions.)
I haven’t specified what close reading is as a practice, but granted that it does pick out something one can do, what you don’t want to do is confuse that practice with the question of how interpretation works—concern (1). The first claim of New Criticism is that there is a practice called “close reading,” a practice that can be differentiated from other practices of reading. To many people, this now seems quite plausible, and the first step towards plausibility is to understand that close reading is a kind of writing.[fn.2] But the New Critics sometimes treated it like simply reading, which makes it more plausible to think that what you are theorizing about is the uptake of conceptual information, i.e. interpretation. When you think of it as a kind of writing, the output of conceptual information, you’re more likely to see what you are doing as offering an interpretation—concern (2). The difference between the two is that when one theorizes about the former—interpretation qua interpretation—the focus is on a general human activity: the uptake of conceptual information is something everyone does all the time. One interprets, in this sense, a painting, the newspaper, or a stop sign as much as Hawthorne or Melville. This kind of theorizing will, indeed, be about the metaphysical necessity of how, exactly, one processes concepts and uses them in communication. As concern (1), put into full swing by Kant, philosophers of language from Austin to Brandom to Habermas have a lot that is relevant to say, but literary theorists have also said many relevant things (from Hirsch to Eco to Fish). However, what one should not do—to repeat—is make the mistake Fish acknowledges having made in his early, career-making books Surprised by Sin and Self-Consuming Artifacts: to confuse a reading with reading.[fn.3]
The only way to glue together concern (1) and (2) is with (3): foundationalism. This is how Descartes gets from the New Science to knowing in general (the logical positivists, too) and it’s the only way to get from a practice like close reading to reading in general. What you do is take one practice and claim that it is the foundation-practice upon which all others rest. This is close to what Kant does, but the transcendental maneuver is slightly different: it doesn’t take an object and “privilege it” (in a phrase still often used for antifoundationalist purposes), it asks what makes the object possible. It is essentially a mechanical question: what needs to be the case for this thing to work as it does? Close reading privileges, in the dirty sense, one kind of literary text: short ones. However, the practice of close reading can be used on anything, it’s just easier on short stuff. What made the New Criticism’s practice of close reading foundationalist is that they first gave a bad answer to how interpretation works (the central move is that it works on a text that is fundamentally isolated from the world—much easier to do on a lyric poem than a historical novel) and then claimed that any other kind of interpretation (e.g., through the use of biography, myth, class consciousness, etc.) is not true interpretation (usually qualified in some way to make “true interpretation” a gateway to the other, now extraneous activities—the other readings aren’t fake, they’re just irrelevant).
What happens when you get rid of foundationalism? Apparently what happens is that you spend another 50 years trying to figure out what getting rid of it means. The Continental philosophy imported in the 60s was all resoundly antifoundational in tenor, yet many foundational-esque habits continued. Some, like Fredric Jameson, tried to be upfront about it. But post-Foucauldian theory—which is something like a confluence of the political sensibilities of Marx with the antifoundational tendencies of Derrida—has struggled, to my mind, in shrugging off foundationalist theory because it feels the impulse to amp up the moral claims of “we should read this way” to “we need, nay, must read this way.” If you read “must” expressively, there’s no problem, but often the “must” comes with a battery of theoretical concerns about the nature of representation. “Representation is socially constructed,” we hear over and over again well after it was something that titillated us, “and therefore we must practice deconstructive criticism because showing how meaning is impossible is the only way to short-circuit the hegemony of European power.” If the vague reference to European power seems a stretch, all you need to do is remind yourself that “semantic authority” over descriptions that pass as true—and that passing as true is the only kind of true you get in this life—is what is produced in what we call “education.” And showing that meaning is impossible would indeed short-circuit such semantic authority (ignoring for the moment how one would have understood the meaning of that sentence). But it seems like a non sequitor from “this is the way representation works” to “we have to read this way.” It is basic to the is/ought dichotomy that Hume pointed out, and if you feel the need, as any form of antidualist, to blur that distinction, if you are also a critic of bourgeois liberalism and its ideology, then you’ll want to remind yourself that blurring that distinction is exactly how the liberal ideologists used their so-called essentialism to move from “I think liberalism is good” to “my cultural representations are the way the world works so I need to destroy everyone else’s.” The movement from is to ought has to be treated as a conversational non sequitor for the very idea of progressive change to be possible. If the necessity was understood to be a purely moral concern, a concern about how the practice of literary criticism should move forward, then the theorist would understand that repeating the lesson about how representation works is not an argument directly for the practice of deconstruction, but merely the preparation for clearing the field of old foundationalist dogma. Once the dogmas are clear, there are still a lot more questions about how one should read—or rather, what kind of readings one should offer.
The lesson I’ve tried to articulate in the above is that if you keep “how does interpretation work?” separate from “what interpretation should I offer?” you will find yourself in the proper disciplinary province of literary theory—reflection about the tools of literary criticism. Having some kind of idea about how interpretation works is, in this understanding, philosophical because interpreting linguistic phenomena is a general human activity and not distinctively literary-critical. This is not to say that philosophers are better at understanding interpretation qua interpretation than literary critics, but the questions and problems peculiar to it (as opposed to problems peculiar to interpreting Hawthorne or Melville) come up with more disciplinary regularity in philosophy than literary criticism (if only because the philosopher spends more time focused just on those abstract questions, and if only because all the special disciplines have forced such focus on the philosopher by making special disciplines around all the more concrete questions). Literary theory should be about what kind of interpretations we should be offering, but they shouldn’t turn on notions of metaphysical or epistemological necessity, for the only kind of reading those kinds of general necessity motivate are compositional—questions about rhetoric and clarity and such. Important questions, to be sure, and part of the province of an English instructor, but not special to literary criticism. Once we scrub Marx of his “science” and Derrida of “one cannot escape the discourse of philosophy,” then what we are left with are a number of tools—tools that can be refined by further reflection—and questions about what purposes we would like to fulfill by forwarding such-and-such a reading.
 My treatment of literary theory here, of course, is very schematic and far too generalized to count as good history. Still the best single account of how we get from the beginning of the 20th century to the end is Vincent Leitch’s American Literary Criticism from the 30s to the 80s (the central movement with the aid of a little imaginative extrapolation), but for the purposes of understanding why French and German philosophy caught on and not, say, John Dewey, Gilbert Ryle, John Austin, or Wittgenstein, Leitch doesn’t make a good answer explicit. Analytic philosophy is introduced almost solely so that one understands that “Continental philosophy” is the other thing that was in fact popular. I don’t think Leitch’s account suffers much at all, but there is a smaller story to be told, one with curious outliers like Hans Frei’s legendary book, The Eclipse of Biblical Narrative, published in 1974. In the preface, Frei acknowledges three major intellectual debts: Erich Auerbach, Karl Barth, and Ryle. The first two are nearly obvious for a book of this sort, but Frei says of Ryle that “anybody interested in hermeneutics has special reason to be grateful to [Ryle’s The Concept of Mind] for its demystification of the concept of intentional personal action, and the author’s steady refusal to divide intelligent activity into separate mental and external components. It is a lesson well applied to the way one views written statements and hence also how to read them” (viii). Frei in fact commends Ryle over the Continentals in doing what he regards as the similar activity of overcoming philosophical dualism. So why did Heidegger and Derrida catch on and not Ryle? Again, there are a few obvious, superficial answers having to do with tone, and Leitch does canvass some of the external motivating factors, but once one gets down to how the superficial facets like tone connect with externalities like the general tenor of American intellectual life and the internal workings of the conceptual dynamics (e.g., what’s attractive about Husserl’s notion of a lebenswelt), as soon as you realize that the conceptual dynamics are more or less the same as those produced somewhere else, you are kicked back to the level of superficialities and externalities. It’s not that superficialities and externalities cannot or do not (or should not) have an impact on the course of history—it’s that what we want out of a good intellectual history is a story that does more than lay down those three different things in parallel tracks and make the gesture “and these three things impacted each other.” I think part of the problem in producing this story is that some are not convinced, as a matter of theory, that the story can or should be told (e.g., Foucauldians) and that for those that are, we still don’t have a simple model of what that kind of historical ligature looks like, and that because—again, at the level of theory—we haven’t a good model of how the material-practical impacts the conceptual-semantic through the mediation of the communal-social (the kind of model people like Ryle and Heidegger were moving toward, and people like Habermas and Brandom have nearly completed).
 For an argument about academic reading generally being a kind of writing, see my "Reading Academically." What is articulated there is a special theory of composition attached to the commitment of writing about other writing (this being analogous to the special theory of relativity's relationship to the general one). What is implicit in this kind of composition (i.e., academic) is a practice of reading that readies the uptake of textual information for the output of textual information. "Close reading" is a further kind of special compositional form that requires something like an analogous special reading form. The best theoretical formulation I've heard articulated about the practice of close reading (by Ed Dryden) is that one takes a block of text and treats it is a synecdoche for the rest of the text. Another way of putting it is that you take one passage from, say, The Scarlet Letter and then treat it as a paradigm for reading the rest of The Scarlet Letter. This allows you to say, "By explaining what this passage means and how it works, I've explained what The Scarlet Letter means and how it works." Clearly, then, this kind of model works really well for when you can fit the whole of a text within the whole of the model: when the part you're treating as a whole (synedoche-style) is actually the whole of it. This is why the New Critics were so fond of lyric poetry. The model becomes attenuated when you apply it to longer texts, but the practice of working out the conceptual dynamics of a single passage (my definition of "close reading") still works well for generating the meaning and mechanics of larger texts, even if these larger texts are more complex because sheer size makes it possible to do more textual things (a fact you can grant even if you wish to argue that poetry is semantically more dense than prose: just think of a novel as a prose-poem). If you want to make a claim that The Scarlet Letter, for example, does three things, the compositional way of arguing for this is to generate each particular thing from three particular emblematic passages that do that thing really well. Once you've generated an understanding of "the thing," you can then apply it at will in writing about other passages of the text, relying on your earlier argued-for-claim-backed-by-evidence. What is important to see, however, is that this is a particular model of reading: you are seeing the text differently because you are looking in the text for something specific—emblematic passages. The ends of writing do alter the means of reading (a point made generally in the opening of "Reading Academically"). This kind of reading doesn't work very well on certain kinds of texts and certain kinds of textual phenomena (e.g., narrative action). Despite the fact that generating textual evidence for a written argument will always look something like close reading, close reading isn't always the best foot forward in reading a text.
 See the introduction to Is There a Text in This Class?, where he describes the evolution in his thinking, from a thesis about Paradise Lost to a full-blown critical methodology to seeing that that wasn’t right as a methodology (though yet as a specific reading) to finally his theory of interpretive communities.