Revision [15107]
This is an old revision of FactcheckingData made by Jamieson on 2015-04-28 12:04:41.
<-- Back to the Speeches Guide
Introduction
Factchecking data is an additional layer of context we add to our Public Statements, meant to measure a politician's honesty. This data is featured in the Political Galaxy app along with Statements.External factcheckers produce factchecks (reports) which analyze politician(s) statements and provide a ruling as to their truthfulness. Vote Smart staff enhance this data and connect these factchecks to Vote Smart content.
Factchecking Data Standards
Scope of Coverage
Statements: all those statements for which a public record can be found, and otherwise meets our criteria for speech collection (including the offices and jurisdictions covered). If the statement does meet our criteria but is not already in our public statements database, these statements should be added.
Factchecks (fact-checking reports): if a report analyzes a politician's honesty in a statement, and that statement meets our criteria for statements, the report will generally be included. Therefore we would exclude:
- reports relating to a statement that doesn't meet our criteria for our inclusion
- reports that solely analyze the consistency of a candidate's position ("flip-flopping"), campaign promise fulfillment, or anything other than the truthfulness of a statement
Core content covered per factcheck: speech_id, factchecker, URL of factchecking report, ruling
Data Sources
Factchecks are sourced from external data partners ("factcheckers"). As of January 2015 this includes the two most-prominent, independent, nonpartisan groups: Factcheck.org FactCheck.org and Politifact
These factchecks are then associated with Project Vote Smart's Public Statements database.
The association of Project Vote Smart's public statements with factchecks are made by Project Vote Smart staff.
Politifact rulings are sourced from Politifact and are not modified. Rulings for Factcheck.org factchecks are assessed individually by Vote Smart staff, based on Factcheck.org's factcheck reports.
Vote Smart staff also flag statements as "questionable" based on rulings from Politifact and Factcheck.org factchecks and in accordance with its internal criteria for rulings.
Criteria for Rulings
A "ruling" is a standardized summary-judgment of a politician's statement attributed to a factchecker.
(note: The numbers reflected below are NOT the same as factcheckruling_id in our database; they were previously using in spreadsheets and are only provided here as a way to understand those spreadsheets)
Project Vote Smart's rulings of Factcheck.org's factchecks:
1 = entirely false
2 = mostly false + some context
3 = 50/50 true/false + lots of additional context
4 = mostly true + some context
5 = entirely true
the following may be seen in previous work, though it does not meet our current criteria for inclusion:
11 = entirely inconsistent or full flip flop
13 = half flip flop
15 = entirely consistent or no flip flop
Politifact's rulings: from their API
Criteria for flagging a statement as "questionable" (this is done through logic and does not involve subjective consideration)
- Factcheck ruling (from Vote Smart) of:
- entirely false
- mostly false + some context
- 50/50 true/false + lots of additional context
- Politifact "statementrulingid" of 3, 4, 5, or 6
Expected Frequency of Updates
Quarterly
Known Issues
Data
- there may be multiple factchecks per speech, possibly from different factcheckers (this also impacts how we might represent and aggregate those rulings, which may vary).
- multiple speeches may be evaluated in one factcheck, and may have different rulings
- currently, factchecks are only related to speech_id's and finer grain associations may be needed- rulings are typically based on a subset of a speech and not the whole speech. Sometimes these quotes might be repeated in multiple statements. This subset may only a subset of the categories or tags of the speech (for example, a debate transcript or State of the Union). In the case of a debate, there would be multiple speakers in one piece of raw evidence, but the factchecking report may only address the comments of one speaker. We have previously discussed using "speech snippets" as a solution but never implemented it. Note: in the first round of dealing with historical data, we did save additional, finer-grain detail, including: categorization of the factcheck report, tagging of the factcheck report, and matching to candidate Ids- this may be useful in the future.
- Currently we are only incorporating factchecks that can be associated with our public statements database. This leaves out factchecks associated with bios or other data types we cover (a rare occurrence, but some examples: 1, 2, 3), factchecks for candidates we normally cover but don't cover speeches for, and speeches that don't meet our criteria for inclusion (like speeches without a direct quote from a candidate. Should we add a way to connect factchecks with bios or other data types? Should we start including non-traditional evidence, like campaign ads or mailers from the candidate's campaign, or social media? Should we change our speech collection criteria, at least for those with associated factchecks? If the raw evidence is no longer publicly available from any source, should we just add the quote being referenced? If not directly tied to a speech or piece of evidence in our database, should we simply add the factcheck and add a way to reference the raw evidence?
- Should we extract Factcheck.org's rulings and keep these raw rulings in our database as well? Factcheck doesn't use a set system but summary judgments could be extracted, like “whopper”
- Should Factcheckers be added as SIGs and connected that way? Arguments for (Kristen, National Director): There are some current SIGs that use candidate statements (often candidate surveys) to evaluate that politician's alignment with them on the issues, but we don't track that, and it's a little different. Also, I have been trying to get factcheck and politifact's evaluations into an aggregate score for use as a rating. If so, can we add a SIG type to distinguish media/journalism groups from special interests? In this scenario, rulings made by Factcheckers could be “SIG Ratings”, with the standardized rulings made by PVS as “Our Ratings.” We may ultimately want to evaluate piece of evidence for a candidate(s) on other dimensions and from other groups- for example, SIG's preferred bill positions (which may also tie into their aggregate ratings); Aggregate of judgments weighed upon a candidate by a SIG should translate into an overall rating. Arguments against (Clinton, Former IT Director): I'm not seeing a reason to treat fact checking orgs as sigs unless there's more of an overlap in what is rated.
- Other associations we might make with our data: Some factchecks refer to statements about issues, about their voting records, about a bill, about their bio, about other candidates, etc. I could see value in the statement being evaluated connected to the aspect of their record they are talking about, but not necessarily the factchecker judgment being directly connected to that aspect of their record. It should also be noted that Vote Smart is sometimes cited as a source for the factcheckers' analysis. I'm not sure there's anything to do with that information.
Representation of factchecks
- define “factchecks” better- specify in some way the dimension being evaluated (in this case, honesty vs. misrepresentation or the factual nature of a statement)
- We have established criteria for “questionable” but what do we do with the rest of our statements? Should we assume truth if not otherwise verified by a Factchecker? For those statements deemed to be true, should we represent that they have been “certified true” or “verified” much like VoteEasy research certified answers?
Key Tasks
The key task is to associate the following content:
- the evidence being evaluated (Vote Smart's public statements)
- the ruling of the evidence made by a fact checker (see: data sources)
- the factcheck report (analysis) of the evidence by the factchecker
In the past, Clinton (former IT Director) was able to scrape factcheck and use Politifact's API to provide CSVs. Among other data, he included the factcheck URLs, which staff would then match to speech_ids and resubmit. The plan at that time was to provide these CSVs on a monthly basis. With an IT backlog, Research staff began manually retrieving URLs from Politifact and Factcheck's respective websites.
To associate factchecking data to statements:
- evaluate if the factcheck report meets our criteria for inclusion (in our experience, this has been about 50%). If it doesn't fit into our normal criteria for inclusion or is a new kind of evidence, and you think it ought to be added, run it by the National Director.
- Find the quote being evaluated in Vote Smart's Public Statements database. Add the public statement if necessary.
- Relate the public statement to the factcheck in accordance with current procedure
- Assign a ruling if not provided clearly by the fact checker (note: this is currently done in bulk for Politifact and individually for Factcheck.org.)
Special situations:
- a factcheck evaluating a statement by Candidate A about something Candidate B said should not be related to Candidate B but should be related to Candidate A
Management of Factchecking Updates
Key Objectives:
Get our data as up-to-date as possible
Work with IT on the following:
- improve process so that updates of this content can be done more frequently
- address "Known Issues" as needed
- add way to input this data to Admin- this may include: adding word search capabilities for public statements; ability to associate factchecks with candidate speeches; separate section for factcheck entries including the ability to browse existing factchecks
- integrate Factcheck.org's API
- integrate data into our other web properties
- integrate data into our public API
- future development (See "Incorporating Fact-Checking Data" and "factchecking data to include" documents on the public drive->cross-department projects->possible future projects)
Pace Estimates (for associating factchecks to public statements):
beginners: recorded at 16-28 factcheck articles/hour using spreadsheet imports (sample size of 2 staff members; this includes approximately 50% of those articles marked "intentionally blank" because they were not factchecking federal active officials)
API/Display in Galaxy
As of early 2014, the following content was fed through the public statements call of Version 2 of the API to the Galaxy app. It is Kristen's understanding that "ruling" was replaced with a flag of "questionable" (this should be verified):
Example snippet :
"factchecks": [
{
"http://www.factcheck.org/2012/10/whoppers-of-2012-final-edition/",
"http://www.factcheck.org/2012/10/dubious-denver-debate-declarations/",
"ruling": "mostly false + some context"
},Our intention is to highlight the statements made by a candidate that were determined to be questionable or some degree of false. So, claims that were determined to be true would be excluded from the current display. This content is highlighted in two places in Galaxy:
- When a politician and issue category or selected, a red star will appear next to "Statements" if at least one of those statements in that category are determined by a factchecker to be questionable
- When the user selects a politician's statements, any statements with a factcheck that has a flag of questionable will display with a red banner that says "Factcheck." Selecting the banner redirects the user to the factchecking report on the factchecker's website
CategoryResearch
