A Proud Member of the Reality-Based Community

Acknowledgements

SEARCH eRiposte!

MEDIA BIAS : COMMENTARY 

This page has been updated and moved to the following website 
(click on the link below or wait to be automatically redirected)

http://media.eriposte.com/2-9.htm

[NOTE: The version below is outdated]

A RESPONSE TO THE PAPER by Tim Groseclose (UCLA/Stanford) and Jeff Milyo (U. Chicago) titled "A Measure of Media Bias" 

by eRiposte

3/13/05 

PREFACE

Last night, as I was Googling for links on media and economic bias, I came across some links which eventually led me to a Language Log post featuring a response by Gloseclose-Milyo to a critique of their paper which purports to identify media bias and the ideology of think-tanks using an interesting approach. It appears this study has generated sufficient interest that Mark Liberman at Language Log has pointed out:

Groseclose and Milyo's study has been approvingly cited by Bruce Bartlett in National Review, by Linda Seebach in the Rocky Mountain News, and by Harvard economist Robert J. Barro in Business Week,  not to mention conservative bloggers like Instapundit, Andrew Sullivan, and Matt Drudge, among a number of others, who trumpet its "objectivity."

I very briefly skimmed through the critique of the paper by Geoffrey Nunberg and Gloseclose-Milyo's (referred to as G-M hereafter) response to the critique. However, I have spent very little time analyzing Nunberg's critique and G-M's response because I wanted to really read the original paper first, to understand the methodology and the findings. In the following, therefore, I summarize my own critique of the paper. [Let me also state that I believe this paper is interesting and my critique is provided dispassionately and without any assumption that there was any partisan agenda behind the paper].

For consistency, I use the word think-tank in this page in the same sense in which the authors use it - to describe not just traditional think-tanks, but advocacy groups as well. This is a debatable point, but it is irrelevant to this response.

SUMMARY

A paper by Gloseclose and Milyo attempts to assess media bias using an approach based on a combination of adjusted ADA scores to assess legislator ideology, and a comparison of think-tank citations by legislators and news media to derive the media's "bias". Based on their methodology (presented and discussed in this paper), they claim that "...contrary to the views and evidence cited above, we find a significant liberal bias in our sample of media outlets." They also claim, for instance, that their data shows the Brookings Institution to be left-leaning and not centrist. 

In this critique I examine the paper from three perspectives:
1. Is the methodology used for assessing the ideology of think-tanks correct and reliable? (Section 1)
2. Is the methodology used for assessing the ideology of the media correct and reliable? (Section 2)
3. Is the definition of media bias used by the authors correct and reliable? (Section 3)

I find that the answers to each of those questions is NO. 

The methodology used by the authors for assessing think-tank ideology is deeply flawed because it omits public or private disagreements that legislators have with think-tanks and does not account for the fact that legislators may agree with think-tanks but not state it publicly for various reasons. This can effectively skew their results in the wrong direction, to an unknown degree. Indeed, the fact that their methodology found the ACLU to be "conservative" was a result of this flaw. 

The methodology used by the authors for assessing media ideology is completely untenable. There are three principal reasons for this:

(a) The approach they use establishes media ideology indirectly, by using the media's think-tank citations and comparing that to think-tank citations by legislators in order to find the legislator whose citations are the closest match. Thus, if a legislator is liberal and the media's think-tank citations match that of the liberal legislator, they would declare the media to be liberal. Setting aside the fact that this definition of media bias is itself incorrect, their claim would be true only if it can be independently proven that the think-tanks cited by the liberal legislator are actually liberal. Their study does not prove this at all, considering that their methodology to establish think-tank ideology is itself deficient.

(b) They use the Median adjusted ADA score for the House to determine the "center" for defining who is liberal and who is conservative. As I show, the median approach is fatally flawed and can dramatically, and incorrectly, skew the derived ideology of the media in a direction that is opposed to the ideology of the majority party in Congress.

(c) The use of the Mean adjusted ADA score for the House is slightly more meaningful than the Median, but even this is completely deficient and incorrect because the ideological center is set not using an independent, objective measure of ideology but based on the positions of the people in the House at a given point in time. The core of the problem here is that their model simultaneously assumes that ADA scores can provide an absolute picture of a legislator's ideology, but that media- and think-tank ideology should be determined not using the same absolute reference but a relative, moving reference that is highly dependent on who's the majority in Congress and what their ideological position is relative to the minority. This is not an acceptable model, for, if the minority party becomes the majority party in the next election, think-tanks or the media would suddenly appear to have switched ideologies even though their positions underwent ZERO change.  

The final, and perhaps most serious, problem with their analysis is their attempt to derive a conclusion of media bias using this study. Their confident conclusion that they have proven "liberal media" bias is simply wrong because the study does not examine anything about whether the media actually reports the positions of liberals or conservatives accurately. Citing a think-tank says nothing about whether that think-tank is accurate or not. And it certainly says nothing about what the media communicates to the viewers when it is not citing think-tanks, which is a big chunk of the time. 

When controlled for other factors (see Appendix A), the more fundamental determinant of bias in news reporting is accuracy -- not whom the news reports cite*. To the extent that news reporting could become inaccurate by citing certain think-tanks over others, one may have a case that think-tank citations could influence the accuracy of the reports. But, G-M have fallen into the trap of assuming that the part is the whole. Think-tank citations are merely one part of the whole - which is the media's accuracy in news reporting. 
[* edited to make sentence less laborious and clearer]

An interesting side note:
There is only one part of their paper which may have some merit (and I say this with qualifications because I cannot be completely sure whether even this is true). This part is their methodology for assessing the ideology of legislators in Congress, wherein they derive an adjusted ADA score (based on Americans for Democratic Action ratings) for each legislator. If the ADA score is actually a good *and* broad representation of the legislators' ideology, the findings of these authors would indicate that in the period of 1993-1999, Democrats in Congress were far more centrist than Republicans in Congress and that Democrats in Congress were far less liberal than Republicans were conservative. This appears to be broadly consistent with the known ideology of Congress at that time and would be an independent confirmation that Democrats are certainly not "too liberal" in this country.

DISCUSSION

SECTION 1. ASSESSING THE IDEOLOGY OF THINK TANKS

SECTION 2. ASSESSING THE IDEOLOGY OF MEDIA OUTLETS 

SECTION 3. DEFINING MEDIA BIAS 

APPENDIX A: Clarifying comments on Section 3

SECTION 1. ASSESSING THE IDEOLOGY OF THINK TANKS

1.1 The authors' state in page 3 of the paper that:

A feature of our method is that it does not require us to make a subjective assessment of how liberal or conservative a think tank is. That is, for instance, we do we need [sic; this is a very critical phrase and unfortunately there is a typo or grammatical error here which makes it's meaning unclear. Based on the content of the rest of the paragraph I infer that the authors mean to say "we do NOT need" and my discussion below is based on this understanding] to read policy reports of the think tank or analyze its position on various issues to determine its ideology. Instead, we simply observe the ADA scores of the members of Congress who cite the think tank. This feature is important, since an active controversy exists whether, e.g., the Brookings Institution or the RAND Corporation is moderate, left-wing, or right-wing. 

They reiterate this with an example in page 7:

The tables shed some light on some much debated topics about the ideological position of various think tanks. First, the table reveals that the position of the Brookings Institution clearly leans left. When we use sentences as our level of observation, the average score of legislators citing Brookings is 50.0, and when we use citations, the average score is 46.2. In contrast, the average score of the House, 44.5, and the average score of the Senate, 40.0, are both more right wing than the average legislator citing Brookings.⁷

Although the authors state that their method is such that it "does not require us to make a subjective assessment of how liberal or conservative a think tank is", their method is in fact making a highly subjective assessment. Why?

• Firstly, by definition, an objective assessment is one in which we would need to actually read through the think tanks policy briefs and compare the details in those briefs to a fixed definition of what is considered liberal, centrist or conservative (one can pick a reference like the ADA if one wishes, but the reference needs to be fixed). That is not being done here because the think tanks ideology is being derived by using a (weighted) metric of who cites the think tank and how often. That metric is a function of:
  • the actual policy issues that are being debated or discussed publicly (e.g., if only 3, out of say 20, policy issues were discussed prominently during the time period of the study it could skew the results) 
  • the legislator citing or not citing the think tank 
  • the assumption that everyone in the sample knows about the think tank as well as its detailed positions on matters of interest to them, among other things
    • In my view, the latter is unlikely to be true for any think tank across the board, and even if true it is likely to be the case for no more than 1 or 2 top think-tanks (although experience has made it clear to me this is exceedingly unlikely. I know of no legislator who is fully familiar with all the key details of even one think tank's policy proposals on every topic of interest to them, let alone multiple). 
• Secondly, while the authors are free to use their current approach, it is, in fact, subjective because it is very unlikely that legislators who cite the think tank agree with everything the think tank says or stands for. For example, some legislators may cite it because their position is in agreement with, say, only one or two or three of the think tank's positions and they may cite it for that reason, repeatedly. Indeed, the authors' own ACLU example directly illustrates this problem. ACLU appears more conservative than it is (according to the authors) because a conservative cited it repeatedly on one specific point of agreement, even though the same conservative likely disagreed with ACLU on most other issues. While the authors did the right thing by pointing this out, this is a significant and fundamental problem that severely impacts the rest of their study and conclusions. Why? Let me explain.
  • As I said, if many legislators that are citing a think tank (e.g., Brookings) do so only on two or three specific topics (among a wide gamut of topics that would encompass one's definition of ideology), and they actually disagree with Brookings' ideology on many other matters, what options do they have? Well, they can either state their disagreement publicly or keep the disagreement private and never mention it. Let's see how the authors' approach deals with these scenarios. 
  • If a legislator decides to cite Brookings in a negative way, that should actually be considered in the model as one less point of support for Brookings. However, the model actually ignores this data point. As the study says, in page 6: "Also, we omitted the instances where the member of Congress or journalist only cited the think tank so he or she could criticize it or explain why it was wrong. About five percent of the congressional citations and about one percent of the media citations fell into this category." This is a structural flaw in the methodology. While I have no idea how much this flaw actually impacts the final results in this particularly study, it is a flaw that cannot be ignored and the authors would be well advised to fix this flaw, as long as they attempt to derive the ideology of the think tank based on the legislator's citing it.
  • Even more seriously, let's consider the other case where the legislator is actually in disagreement with the think tank on many topics but decides not to cite the disagreement publicly (i.e., he or she keeps it private and doesn't mention the name of the think tank). If this legislator has otherwise mentioned the think tank positively, the method used by the authors would, again, dramatically overstate the think tank's ideology match with the legislator because the legislator's negative views of the think-tank on other occasions is not included in the model (simply because it is private and not publicly known). 
    • These two cases resonate directly with the ACLU case the authors highlight. In that case, the authors have presented the data for ACLU, un-massaged and to address the anomaly of ACLU appearing as a conservative organization they have suggested the following in page 8: "If we omit ACLU citations that are due to McConnell, then the average score, weighted by sentences, increases to 70.12". Unfortunately, that is the wrong approach to take to fix this problem. The way to fix this problem is to actually ADD all those instances in which Republicans actually disagreed with ACLU, not incorrectly and artificially remove situations where *they agreed* in order to get an average score that seems more in sync with a *separately established* reality. In other words, if we already knew ACLU is "liberal" and need to know that to "adjust the data", then what is the value or point of this study? (NOTE: I am actually thankful to the authors for having been honest enough to provide the ACLU example because I would have otherwise had to find a hypothetical example to illustrate my thought process. So my comment here is purely from an analytical viewpoint and no harsh criticism is intended.)
      • The ACLU case may seem extreme, but it is not. It is not at all difficult to fathom a situation where support for a think tank (say, Brookings) is overstated with certain (e.g., left-leaning) legislators because of the factors described above.
  • Moreover, as I said earlier, a legislator may cite a think tank not because he or she mostly agrees with the think tank but because that think tank's view is closer to his or her view than any other think-tank the legislator is aware of or cares to cite, e.g., say, a legislator likes MoveOn.org but fears the negative publicity he may get from his constituents through fake attack ads in the next election. Should he cite MoveOn's position or try and pick a more centrist think-tank like Brookings that is somewhat close to his position, to make his point? Obviously some of these think-tanks do overlap in some of their positions (e.g., "No Crisis in Social Security"). This is yet another example, where the authors' model would overstate the think tank's ideology match with the legislator. 
• The cases I have provided, illustrate, a very significant structural problem with the authors' model. Let me also add that there is an additional problem inherent in what I discussed above. A legislator may have a certain policy position but rarely cite a think tank that agrees with that position publicly, simply because (a) he or she may not be aware of the think tank itself, (b) he or she may not even be aware that the think tank's position is actually in sync with his or her position or (c) he or she may NOT find it necessary to cite the think tank to make a point. All of this would understate the think-tank's ideological similarities with the legislator. This could cut both  ways in terms of the impact to the final results.

Thus, the very basis of this study - deriving think-tank ideology in a relative fashion (rather than objective fashion) - causes its results to be highly questionable. While it is clear that a lot of thought went into it and the idea pursued here is interesting, the basic assumptions required to make this model work are a fundamental limitation in being able to derive results that are even reasonably accurate. The net effect of the model could be to artificially push the results in the direction of making the think tank far closer to the legislator's ideology when that is not true in reality (it is also possible to mask closer ideological matching with lesser known or more ideological think-tanks which exist in reality). Scientifically speaking, and with due respect, this flaw is so severe that I simply cannot accept the conclusions of this paper on the derived ideology of the think-tanks. 

Let me also add that the same arguments apply to the treatment of the media as well, but also in a different sense. It is one thing for a legislator to cite a think-tank approvingly. In a news report, though, unless the media outlet specifically states that it approves the position of the think-tank cited in the news report, assuming that the media outlet somehow overtly shares the ideology of the think-tank in a big, unwarranted leap of faith. For instance, it may reflect an unintentional bias in the choice of think-tank based on the reporter's knowledge base, or a reflection of laziness in reporting, or simply a matter of day-to-day editorial judgment (which has to consider the timeliness of a news story, among other things). Broadly speaking I would tend to agree that the relative proportion of liberal, conservative or centrist think-tanks that a media outlet cites is a piece of information worth knowing. But the methodology used in this particular paper would really not answer this question with reasonable accuracy, since it is flawed. 

1.2 In page 5 of the paper, the authors say that:

We looked for instances where the legislator cited a view or a fact stated by a member of the think tank. We then counted the sentences in the citation. We also recorded the average adjusted ADA score of the member who cited the think tank.³ Along with direct quotes, we sometimes included sentences that were not direct quotes. For instance, many of the citations were cases where a member of Congress noted "This bill is supported by think tank X." Also, members of Congress sometimes insert printed material "into the Record," such as a letter, a newspaper article, or a report. If a think tank was cited in such material or if a think tank member wrote the material, we counted it just as if the member of Congress had read the material in his or her speech. 

We did the same exercise for stories that media outlets report, except with media outlets we did not record an ADA score. Instead, our method estimates such a score. 

Sometimes a legislator or a media outlet noted an action that a think tank had taken—e.g. that it raised a certain amount of money, initiated a boycott, filed a lawsuit, elected new officers, or held its annual convention. We did not record such cases in our data set. However, sometimes in the process of describing such actions, the reporter or member of Congress would quote a member of the think tank, and the quote revealed the think tank’s views on national policy, or the quote stated a fact that is relevant to national policy. If so, we would record that quote in our data set. For instance, suppose a reporter noted "The NAACP has asked its members to boycott businesses in the state of South Carolina. `We are initiating this boycott, because we believe that it is racist to fly the Confederate Flag on the state capitol,’ a leader of the group noted." In this instance, we would count the second sentence that the reporter wrote, but not the first. 

This section raises a question whose answers, again, could dictate the reliability of the results of the study.

Maybe this is mentioned elsewhere in the paper and I missed it, but I don't see a specific qualification associated with a legislator (or news organization) citing a particular think-tank -- in the sense of whether the citation is accompanied by an overt or implied agreement with that think-tank on the particular item being cited. For example, if a legislator cites a think-tank to make a neutral observation (e.g., "Here's what Brookings says, but I don't know whether they are right or not"), how is that counted? In the description in the paper, the authors say: "...member of Congress would quote a member of the think tank, and the quote revealed the think tank’s views on national policy, or the quote stated a fact that is relevant to national policy. If so, we would record that quote in our data set." Does the member of Congress actually have to agree with the quoted portion for it to be recorded?

SECTION 2. ASSESSING THE IDEOLOGY OF MEDIA OUTLETS 

2.1 In page 2 of the paper, the authors say:

To compute our measure, we count the times that a media outlet cites various think tanks. We compare this with the times that members of Congress cite the same think tanks in their speeches on the floor of the House and Senate. By comparing the citation patterns we can construct an ADA score for each media outlet.

If I understand correctly, the authors are assessing "media bias" by extracting adjusted ADA (Americans for Democratic Action) scores for media outlets by comparing their think-tank citations to those of the legislators whose adjusted ADA scores are independently estimated using voting records. Put another way, G-M's goal is to assess the bias of the media outlet relative to that of legislators - using the common variable of think-tank citations.

However, if I understand their methodology correctly, there's an important point of detail which should be noted. 

As they say in pages 2-3:

As a simplified example, imagine that there were only two think tanks, one liberal and one conservative. Suppose that the New York Times cited the liberal think tank twice as often as the conservative one. Our method asks: What is the estimated ADA score of a member of Congress who exhibits the same frequency (2:1) in his or her speeches? This is the score that our method would assign to the New York Times.

This is perhaps the most important portion of their entire paper. Why?

What they are effectively saying is that if the nature of think-tank citations by a legislator who would be considered liberal based on his/her ADA-score, matches the nature of citations by the New York Times, then the New York Times would be deemed liberal (and have a similar ADA score). But is it really that straightforward? Put another way, is there any situation under which this assumption would be wrong? The answer is, resoundingly, yes. Consider a hypothetical (not really, but let's say it is a hypothetical) situation where liberal legislators have a habit of citing centrist (or less likely, conservative) think-tanks. In this scenario, if the New York Times in similar fashion, cites those think-tanks which are centrist (or conservative), then the New York Times will decidedly not be considered liberal - it will have to be considered "centrist" by the authors' own assumption. The problem here is that the think-tank's ideology is NOT assessed independently. 

According to the authors' model, for the media to be considered liberal, the think-tanks cited by liberal legislators need to be established unequivocally as being liberal. I have already shown in Section 1 that the methodology used by the paper to determine the ideology of the think-tanks is fundamentally flawed. So, right away, we can conclude that the main conclusion of this paper - that most of the media outlets they examined are "liberal" when it comes to citing think-tanks - does not hold. 

This is another serious flaw and the final outcome is not surprising at all because this is the kind of difficulty one runs into when one does not assess ideology objectively, but rather, extracts it with reference to something else. 

[NOTE: The above argument is not specific to the liberal perspective. If conservative legislators end up excessively citing centrist or liberal think-tanks (as established using an independent ideology analysis), one would run into the same kind of problem. In other words, this is a structural problem with the model and has nothing to do with partisan or ideological issues].

A couple of additional points: 

• It is not at all implausible that left-leaning legislators may cite more centrist think-tanks in public than progressive/liberal ones (even today). The study reported by the authors was conducted in the period 1993-1999 when Democrats were led by one of the most centrist Democratic Presidents in modern times. In part due to Clinton and the DLC and in part due to pressure from the Republican Party, they were under pressure to appear more centrist because of their congressional losses (partly due to the defeat of the Clinton healthcare plan which was portrayed negatively (and fears of excessive "liberalism" stoked) by the Republicans and the media). This situation has not changed substantially since Bush came to power. Until Jan 2005, the party had been influenced far more heavily by the DLC, especially on economic matters (which is the mainstay of think-tanks like Brookings), but to some extent even on socio-economic issues like Welfare.  
• Even today, the passage of the God-awful Bankruptcy Bill in the Senate is a measure of how legislators in the Democratic party (and Republican party) decided to ignore their real liberal (and conservative) values. Needless to say, overt dalliances with liberal groups are often met with negative publicity in the media due to the Republican Noise Machine. Let me give you another hypothetical example to illustrate what I mean. A liberal senator afraid of being associated with say, MoveOn.org, may over a period of time repeatedly cite Brookings, with whom he only partially agrees with (e.g., let's say for argument, that the legislator, MoveOn.org and Brookings all agree that social security is not "in crisis", but the legislator and MoveOn.org feel that the way to address social security problems is to repeal some of Bush's tax cuts, whereas Brookings may be considering raising the retirement age as a solution (this is PURELY hypothetical and I really don't know Brookings' latest position on social security)). The legislator may simply choose to cite Brookings to make the point that there is no crisis in social security but say nothing about retirement age increases, even though he is against it. This will make Brookings seem more liberal than it is, and likewise make the media outlet seem more liberal if it cited Brookings more. But the reality would be quite different. (This example only goes to illustrate that the method used in this paper is extremely blunt and it is no surprise that unreliable results emerge.) 

2.2 There is another serious problem with the authors' methodology, which represents another fundamental underpinning of this paper. Here, I am referring to the methodology used to extract the reference ADA score, which demarcates "liberals" and "conservatives" in Congress by defining the "center". Clearly, unless this methodology is air-tight, even the definition of which legislator is "liberal" and which legislator is not, would be questionable.

2.2.1 To address this point I would like to first quote the section in their paper that deals with this topic (p 12). 

Digression: Defining the "Center"

In discussing left- or right- wing biases of the media, one should be careful how he or she defines center. We think the most appropriate definition refers to a central voter, as opposed to a central member of Congress. Accordingly, we think that it is more appropriate to compare media scores to the House as opposed to the Senate, since the Senate disproportionately represents small states. Next, we think it is more appropriate to use the median House member, instead of the mean. One reason is that, because of The Median Voter Theorem (Black, 1957), one should expect policy to be at the median instead of the mean. Another reason is that comparisons to a mean can be manipulated by the ADA’s choices of roll call votes, whereas comparisons to a median are not subject to such manipulation.

I'm not an expert on election theory; so I had to look up the "Median Voter Theorem" and a few websites have some commentary on it - here, here, and here. The last of these links, says, for instance: "There is no more transparent nor easily communicated explanation of political outcomes in a democracy than that all political outcomes reflect median voter preferences."

While the median voter model seems like a good concept, it is very clear that it is a theory that is rarely applicable in a political and media environment where the voter is ill-informed and where the politicians are not beholden to the median voter on every issue. For the policy and political outcomes to be where the median voter is, I would assume that at least three conditions need to be satisfied. 

(a) There is such a thing as a median voter 
(In a deeply polarized environment with entrenched beliefs I wonder whether this is a valid assumption, but, for argument's sake, let's assume (a) is true. Then, the following also need to be true...)

(b) The median voter knows which politicians actually represent his or her real needs and votes for/elects them

(c) The elected politicians in Congress actually make compromises to craft policy - as dictated by the position of the median voter

With (b) and (c), the Median Voter Model breaks down in reality. Especially considering this study is of media bias, clearly biases in the media can result in impressions on voters that may sway them in the wrong direction. After all, that is one of the prime reasons to be worried about media bias. Assuming that the voters are all well-informed to know that they are voting for the right candidate is therefore a stretch, when that collides with a simultaneous assumption that there is some media bias. Put another way, median voters may not be electing candidates who actually stand for what the voters think they stand for. [This is not conjecture. During election 2004 many voters were quite ill-informed].

Even when (b) is valid, (c) usually is not. This is also common knowledge. Two ongoing issues serve as good examples. George Bush and the Republican Party said very little about social security privatization during the 2004 campaign. Yet, as a majority party they are pushing for this even though the median voter is clearly against it. Likewise, during the campaign, neither Party really spoke about the egregious Bankruptcy bill that just passed in the Senate. Yet, because the voters do not know the details of this Bill (pro-corporate media bias, anyone?), it is likely to pass even though neither the median nor average voter is likely in favor of it.  [Additionally, and this is a minor point, but are there really median voter policies in situations where Congress is deeply polarized and it is literally one party rule led by the extreme wing of one party?] 

Having said that, G-M are free to pick the median House member to set their center, but other than as a theoretical construct, it is not likely to have much merit from a practical standpoint. That said, this is only the tip of the iceberg for there is a far more fundamental problem with the approach. 

2.2.2 Let me quote the authors again (p 12-13) where they explain their other reason for choosing the median House member to set the ideological "center":

To see this, first note that the ADA has considerable leeway in the roll call votes that it chooses. For instance, suppose it chooses many roll calls such that the cut point of the roll call lies between moderates and extreme liberals. Such a cut point would cause moderates to form a coalition with extreme conservatives on the roll call. (An example of such a roll call would be a bill to ban partial-birth abortions. Here, moderates and conservatives favor the ban, and only extreme liberals oppose it. ) A prevalence of such cut points would cause moderates to have ADA scores more similar to conservatives  than liberals. Meanwhile, if it predominantly chose cutpoints on the other side, then the ADA would cause moderates to have ADA scores more similar to liberals than conservatives. 

Because of this leeway, with one set of roll calls, the ADA could make a member of Congress or media outlet appear more left-wing than the mean score. However, with a different set of roll calls the ADA could make the same member of Congress or the same media outlet appear more right wing than the mean score. To see this, consider the following example. Suppose there are only five members of Congress. The most left-wing legislator is Member 1, who is more left-wing than member 2, who is more left-wing than member 3, and so on. Suppose media outlet A has an ideology identical to member 2. Consequently, its ADA score (that our method estimates) will be identical to member 2’s score (at least in expectation). 

Now suppose that the ADA chooses four roll calls, such that the first roll call has a cut point between members 1 and 2, the second has a cut point between members 2 and 3, and so on. Because the distribution of cut points is uniform, member 1 receives 100 ADA score, member 2 and media outlet A to receive a 75, member 3 receives a 50, and so on. The mean ADA score of the legislators is 50. Thus, this set of roll calls makes media outlet A appear more left-wing than the mean score. 

Next, instead suppose that the ADA chooses four roll calls such that each has a cut point between members 1 and 2. This would cause member 1 to receive a 100 score.  Media outlet A and members 2, 3, 4, and 5 would receive a 0 score. The mean ADA score in this case would be 20. Thus, this set of roll calls makes media outlet A appear more right-wing than the mean score.

Meanwhile, for this example, regardless of the ADA’s choice of cut points, media outlet A’s score will necessarily be greater than or equal to the median’s score (member 3). That is, unlike the case where we use the mean score as a comparison, it is impossible to make media outlet A appear more right-wing than the median score. 

The point of this example is not to suggest that the ADA might intentionally choose roll calls to manipulate a legislator’s or media outlet’s perceived ideology relative to the mean. Rather it is to demonstrate an arbitrariness that exists when one uses a mean score for comparison. The same arbitrariness does not exist with median scores. As a consequence, we think it is appropriate to compare the scores of media outlets with the House median, 39.0.

A careful reading of the example cited by the authors shows that by choosing the median value, they are clearly (perhaps unintentionally) biasing the final result towards the media outlet A being "more liberal" than it really is. In fact, although they claim that they are picking the median because picking the mean would introduce arbitrariness, the reality is exactly the opposite in this example. (This seems strange again, because as I pointed out also in Section 1 while the authors felt they were being objective, rather than subjective, the opposite was true there as well).

This should not be surprising, but this is the built-in result and hazard of using the median approach. What is does is that it takes away any objective definition of ideology and let's ideology be defined by an arbitrary reference that has little or no connection to the objective definition of ideology.

• Let me illustrate this with another example. Let's take the same case the authors have and change it slightly to say the following:
  Suppose that the ADA chooses 10 roll calls such that only 1 has a cut point between members 1 and 2 and the cut point is below member 1 for the remaining 9. This would cause member 1 to receive a score of 10. Media outlet A and members 2, 3, 4, and 5 would receive a 0 score. The mean ADA score in this case would be 2. The median would be 0.
  • Consider the reality here. In this example, ALL members are on average very conservative based on their ADA scores. Yet, in the median approach, media outlet A, while sharing member 2's ADA score of 2 (which would really label it "conservative"), would still be considered "centrist" because its ADA score lines up with the median ADA score for all the members!
• Take another example. 400 members in all. 210 get a 0 on every roll call, and media outlet A shares its ideology with one of those members. The remaining 190 get a 100 on every tool call. (So an imaginary House with 210 arch-conservatives and 190 arch-liberals).
  • Median ADA score = 0
  • Mean ADA score = 47.5.
  • Per Median approach, Media Outlet A ADA score = 0 = "centrist"
  • Per Mean approach, Media Outlet A ADA score = 0 = "conservative" [which is actually the reality]
• Let me point out again that this flaw can cut both ways. If the House is comprised of majority liberals and minority conservatives (flip the numbers in the above example), the median approach would say that even if the media were aligned with one of the arch-liberals, it is "centrist". So, left-leaners should not be the only ones worried about this model! 

BOTTOMLINE: The median approach is fatally flawed and can dramatically, and incorrectly, skew the derived ideology of the media in a direction that is opposed to the ideology of the majority party in Congress. One might therefore be tempted to consider the mean approach rather than the median, but that has its own fatal flaw (below).

2.2.3 The built-in assumption in the study is that the ADA score fully represents the real tenets of liberalism since we are using it to establish how liberal or conservative the legislators are. So, it is therefore strange that the authors abandon the objective definition of the degree of liberalism as defined by the ADA scores to base their judgments of legislator-, think-tank- and media ideology on a relative reference that may not have much to do with the real tenets of the ideology at all. Let me explain.

First, we need to make an assumption here. It needs to be assumed that an ADA score is indeed the correct measure of a legislator's ideology (I have no idea what the ADA's policy positions are and that's why I'm making this point). For the sake of argument let us assume it is.

Now, let's look at Table 2 of the paper:

Table 2. Mean and Median Adjusted ADA Scores, by Chamber and Party,
Averaged over the Period 1993-1999

	House	Senate	Ave. of House and Senate
Republican Mean	11.4	11	11.2
Democrat Mean	76.5	71.7	74.1
Chamber Mean	44.5	40.0	42.2
Chamber Median	39.0	36.9	38.0