Wiki source for FactCheckPoliciesProcedures
@@=====**Fact Checking Policies and Procedures**=====@@
====Introduction====
Fact checks are pieces of data, which are created by 3rd party sources (we use Factcheck.org, Politifact.com, and the Washington Post Fact Checker). These 3rd party sources generate verifying articles (or fact checks) that state whether a politician's statement is true, false, or somewhere in between. (This could relate to the whole statement, or just a small part.) The goal of this page is to inform on how to collect and apply these fact checks to our data. Currently, we only collect and apply fact checks to public statements.
====**Collection**====
The collection of fact checks resembles speech collection, with a few additional elements. We go directly to the **primary source**, which in this case, is Factcheck.org, Politifact.com, and the Washington Post Fact Checker. We also search these websites to find new fact checks that have been published since the last update. From there we enter them into admin. Below outlines the steps to take to ensure accurate and thorough collection:
1) Go to either Factcheck.org, Politifact.com, or washingtonpost.com/news/fact-checker
2) Search the website to find where new content is published (this will be flushed out more specifically by website later).
3) Go through each individual fact check to evaluate its potential for application to our data by:
1) Reading/skimming the fact-check article to find if it mentions a politician and something they said (speech, etc.)
1) If you find a politician and something they said, find the quote of the speech, the date it was given, and then look for the relevant statement in Admin
4) If you find a suitable fact-check article, enter it into admin using the "Fact Check Editor" Module. To do so, you will need to complete the following fields:
1) URL from the fact check article itself
1) Ruling (Entirely True, Mostly True, 50/50 True/False, Mostly False, Entirely False, Pants on Fire!)
1) Speech ID; after you type in the speech_id and search, a list of candidates that are shared on the speech (if applicable), will appear. Select the candidates that the fact check is for. If there is only one candidate in the speech, that will be the only person that shows up.
1) Categories and Tags: these should be specifically about the fact check and the quote, although they will probably overlap with the public statement as well. You may not end up using all categories and tags used on the original statement, and that's OK.
**Note:** If a public statement does not exist in our database, please find and add the public statement. This is especially true for politicians who have not held office for a period of time because we may have stopped tracking them (i.e. Hillary Clinton, Mike Huckabee, etc.). If the statement comes from a source we wouldn't normally collect for, let your director know.
====**Rulings**====
We use a specific set of rulings for all fact checks, regardless of whether the source of the fact checks provides one. We adapted the rulings that Politifact.com produces into a very similar set of rulings so that they may be applied to both Politifact.com and Factcheck.org. Below are the rulings that we use and a description of them, for the instances where we need to manually determine a ruling:
1) Entirely True (ID = 5) or the Geppetto Checkmark (ID = 29): this statement was determined to be accurate, and without any falsehoods.
1) Mostly True + some context (ID = 4) or One Pinocchio (ID = 24): this statement had many true elements to it, however, contained falsehoods, which was material evidence in the article.
1) 50/50 True/False + lots of additional context (ID = 3) or Two Pinocchios (ID = 25): this statement is neither mostly true or mostly false, and provides material evidence for both sides of the argument for true or false.
1) Mostly False + some context (ID = 2) or Three Pinocchios (ID = 26): the statement had many false elements to it, however, contained some truth that is material evidence in the article.
1) Entirely False (ID = 1) or Four Pinocchios (ID = 27): the statement was determined to be inaccurate, and without any truth.
1) Pants on Fire (ID = 23): only use for politifact articles
1) Bottomless Pinocchio (ID = 28): Only use for Washington Post articles. From WaPo's site: "The bar for the Bottomless Pinocchio is high: Claims must have received Three or Four Pinocchios from The Fact Checker, and they must have been repeated at least 20 times."
====**Current Sources of Fact Checks**====
===Politifact.com===
==Where to Look?==
Politifact.com publishes all of their fact checks under the "Truth-O-Meter" tab and the "All" option. This will contain all the fact checks that we would like to collect, however, it is always a good idea to do a quick scan of the website for additional publishing locations or new types of content.
==Rulings==
Politifact.com assigns their fact checks rulings (True, Mostly True, Half True, Mostly False, and Pants on Fire). We have adapted those ruling so that we can use them for both Politifact and Factcheck. The rulings we use are outlined above, as well as in the Factcheck.org section of this page. Regardless of any differences, Politifact's rulings align perfectly with the rulings we use (we based our rulings off of theirs).
===Factcheck.org (as of June 2015)===
==Where to look?==
Factcheck.org has one place where you should begin searching for fact checks. Under the "Articles" tab on the top toolbar, there is a link to "The Wire." This is where all fact checks produce by FactCheck.org should be published. With that being said, it is always helpful to do a quick scan of the website for additional publishing locations or new types of content.
==Rulings==
Currently, Factcheck.org does not provide or determine a ruling. Instead, whoever is inputting the fact check needs to manually designate an appropriate ruling for that fact check. To do this, please evaluate the fact check only and base your ruling off of what you believe Factcheck.org would have given if they determined rulings. The ruling system we use for Factcheck.org is nearly identical, with the exception of "Pants on Fire." Do not apply "Pants on Fire" to any fact check from Factcheck.org, however, you are free to use Entirely True, Mostly True, 50/50 True False, Mostly False, and Entirely False.
To make a ruling for a fact check from Factcheck.org, use your best logical ruling and consult others if you have ANY question about which ruling to use.
===Washington Post Fact Checker (as of October 2019)===
==Where to look?==
Go to washingtonpost.com/news/fact-checker to view the latest fact-checks from the Washington Post. This is a running list of articles. This is the best place to look. At this time, it's not possible to filter by politician, so it's best to keep up to date on Washington Post fact checks. Dealing with this page when you have a backlog is not fun. There is an archive option (top right, next to the Fact Checker header) where you can filter by month and year.
==Rulings==
The Washington Post Fact Checker uses the Pinocchio Test, which assigns either a Geppetto Checkmark (for entirely true statements) or 1-4 Pinocchios. You can read about these rulings [[https://www.washingtonpost.com/politics/2019/01/07/about-fact-checker/ here]]. If you don't see a ruling, consult your director.
====**Other Factors to be Aware of**====
~- Multiple speeches may be evaluated in one factcheck, and may have different rulings. In the case of multiple rulings, please make multiple entries.
~- If a public statement does not exist, do your best to find and add this statement.
~- If a fact check report does not meet our criteria for inclusion, but you feel that it should, please consult with your director.
~- For a sample size of 2 staff members, the pace of collection and entry (using a spreadsheet) was 16-28 fact checks per hour. This needs to be flushed out more, however, can be used as a baseline.
~- Currently, only false fact checks are //displayed//, however, we still need to collect true fact checks so that we can display them in the future.
====Introduction====
Fact checks are pieces of data, which are created by 3rd party sources (we use Factcheck.org, Politifact.com, and the Washington Post Fact Checker). These 3rd party sources generate verifying articles (or fact checks) that state whether a politician's statement is true, false, or somewhere in between. (This could relate to the whole statement, or just a small part.) The goal of this page is to inform on how to collect and apply these fact checks to our data. Currently, we only collect and apply fact checks to public statements.
====**Collection**====
The collection of fact checks resembles speech collection, with a few additional elements. We go directly to the **primary source**, which in this case, is Factcheck.org, Politifact.com, and the Washington Post Fact Checker. We also search these websites to find new fact checks that have been published since the last update. From there we enter them into admin. Below outlines the steps to take to ensure accurate and thorough collection:
1) Go to either Factcheck.org, Politifact.com, or washingtonpost.com/news/fact-checker
2) Search the website to find where new content is published (this will be flushed out more specifically by website later).
3) Go through each individual fact check to evaluate its potential for application to our data by:
1) Reading/skimming the fact-check article to find if it mentions a politician and something they said (speech, etc.)
1) If you find a politician and something they said, find the quote of the speech, the date it was given, and then look for the relevant statement in Admin
4) If you find a suitable fact-check article, enter it into admin using the "Fact Check Editor" Module. To do so, you will need to complete the following fields:
1) URL from the fact check article itself
1) Ruling (Entirely True, Mostly True, 50/50 True/False, Mostly False, Entirely False, Pants on Fire!)
1) Speech ID; after you type in the speech_id and search, a list of candidates that are shared on the speech (if applicable), will appear. Select the candidates that the fact check is for. If there is only one candidate in the speech, that will be the only person that shows up.
1) Categories and Tags: these should be specifically about the fact check and the quote, although they will probably overlap with the public statement as well. You may not end up using all categories and tags used on the original statement, and that's OK.
**Note:** If a public statement does not exist in our database, please find and add the public statement. This is especially true for politicians who have not held office for a period of time because we may have stopped tracking them (i.e. Hillary Clinton, Mike Huckabee, etc.). If the statement comes from a source we wouldn't normally collect for, let your director know.
====**Rulings**====
We use a specific set of rulings for all fact checks, regardless of whether the source of the fact checks provides one. We adapted the rulings that Politifact.com produces into a very similar set of rulings so that they may be applied to both Politifact.com and Factcheck.org. Below are the rulings that we use and a description of them, for the instances where we need to manually determine a ruling:
1) Entirely True (ID = 5) or the Geppetto Checkmark (ID = 29): this statement was determined to be accurate, and without any falsehoods.
1) Mostly True + some context (ID = 4) or One Pinocchio (ID = 24): this statement had many true elements to it, however, contained falsehoods, which was material evidence in the article.
1) 50/50 True/False + lots of additional context (ID = 3) or Two Pinocchios (ID = 25): this statement is neither mostly true or mostly false, and provides material evidence for both sides of the argument for true or false.
1) Mostly False + some context (ID = 2) or Three Pinocchios (ID = 26): the statement had many false elements to it, however, contained some truth that is material evidence in the article.
1) Entirely False (ID = 1) or Four Pinocchios (ID = 27): the statement was determined to be inaccurate, and without any truth.
1) Pants on Fire (ID = 23): only use for politifact articles
1) Bottomless Pinocchio (ID = 28): Only use for Washington Post articles. From WaPo's site: "The bar for the Bottomless Pinocchio is high: Claims must have received Three or Four Pinocchios from The Fact Checker, and they must have been repeated at least 20 times."
====**Current Sources of Fact Checks**====
===Politifact.com===
==Where to Look?==
Politifact.com publishes all of their fact checks under the "Truth-O-Meter" tab and the "All" option. This will contain all the fact checks that we would like to collect, however, it is always a good idea to do a quick scan of the website for additional publishing locations or new types of content.
==Rulings==
Politifact.com assigns their fact checks rulings (True, Mostly True, Half True, Mostly False, and Pants on Fire). We have adapted those ruling so that we can use them for both Politifact and Factcheck. The rulings we use are outlined above, as well as in the Factcheck.org section of this page. Regardless of any differences, Politifact's rulings align perfectly with the rulings we use (we based our rulings off of theirs).
===Factcheck.org (as of June 2015)===
==Where to look?==
Factcheck.org has one place where you should begin searching for fact checks. Under the "Articles" tab on the top toolbar, there is a link to "The Wire." This is where all fact checks produce by FactCheck.org should be published. With that being said, it is always helpful to do a quick scan of the website for additional publishing locations or new types of content.
==Rulings==
Currently, Factcheck.org does not provide or determine a ruling. Instead, whoever is inputting the fact check needs to manually designate an appropriate ruling for that fact check. To do this, please evaluate the fact check only and base your ruling off of what you believe Factcheck.org would have given if they determined rulings. The ruling system we use for Factcheck.org is nearly identical, with the exception of "Pants on Fire." Do not apply "Pants on Fire" to any fact check from Factcheck.org, however, you are free to use Entirely True, Mostly True, 50/50 True False, Mostly False, and Entirely False.
To make a ruling for a fact check from Factcheck.org, use your best logical ruling and consult others if you have ANY question about which ruling to use.
===Washington Post Fact Checker (as of October 2019)===
==Where to look?==
Go to washingtonpost.com/news/fact-checker to view the latest fact-checks from the Washington Post. This is a running list of articles. This is the best place to look. At this time, it's not possible to filter by politician, so it's best to keep up to date on Washington Post fact checks. Dealing with this page when you have a backlog is not fun. There is an archive option (top right, next to the Fact Checker header) where you can filter by month and year.
==Rulings==
The Washington Post Fact Checker uses the Pinocchio Test, which assigns either a Geppetto Checkmark (for entirely true statements) or 1-4 Pinocchios. You can read about these rulings [[https://www.washingtonpost.com/politics/2019/01/07/about-fact-checker/ here]]. If you don't see a ruling, consult your director.
====**Other Factors to be Aware of**====
~- Multiple speeches may be evaluated in one factcheck, and may have different rulings. In the case of multiple rulings, please make multiple entries.
~- If a public statement does not exist, do your best to find and add this statement.
~- If a fact check report does not meet our criteria for inclusion, but you feel that it should, please consult with your director.
~- For a sample size of 2 staff members, the pace of collection and entry (using a spreadsheet) was 16-28 fact checks per hour. This needs to be flushed out more, however, can be used as a baseline.
~- Currently, only false fact checks are //displayed//, however, we still need to collect true fact checks so that we can display them in the future.
