Scholarly article on topic 'Improving the Nutritional Impact of the Supplemental Nutrition Assistance Program:'

Improving the Nutritional Impact of the Supplemental Nutrition Assistance Program: Academic research paper on "Health sciences"

CC BY-NC-ND
0
0
Share paper
OECD Field of science
Keywords
{}

Abstract of research paper on Health sciences, author of scientific article — Cindy W. Leung, Aviva A. Musicus, Walter C. Willett, Eric B. Rimm

Introduction The Supplemental Nutrition Assistance Program (SNAP) is the largest federal food assistance program designed to alleviate food insecurity and improve dietary intake. This study assessed the opinions of SNAP participants and food-insufficient nonparticipants on their perceptions of the program and strategies to improve its nutritional impact. Methods This study surveyed 387 individuals via Amazon Mechanical Turk, of whom 118 were SNAP participants and 269 were food insufficient but not enrolled in SNAP (nonparticipants). Open-ended questions were coded and analyzed for thematic content. For closed-ended questions, response frequencies were compared using chi-square tests. Data were analyzed in 2016. Results SNAP participants reported that the program successfully served its primary purpose: to allow individuals to buy enough food to make ends meet and reduce food insecurity. Importance was placed on buying food for their children/families and the ability to allocate money for other expenses. To improve the nutritional impact, SNAP participants suggested more nutrition education, increasing the benefit allotment, incentivizing healthful foods, and excluding unhealthful foods for purchase with SNAP. When participants and nonparticipants were asked to choose between SNAP and a nutritionally enhanced program combining healthy incentives with exclusions for sugary beverages (i.e., SNAP+), 68% of participants and 83% of nonparticipants chose SNAP+. Of those who initially chose SNAP, 68% of participants and 64% of nonparticipants chose SNAP+ if paired with a 50% increase in total benefits. Conclusions SNAP participants and food-insufficient nonparticipants support policies that facilitate purchases of healthful foods and limit purchases of unhealthful foods, specifically sugary beverages.

Academic research paper on topic "Improving the Nutritional Impact of the Supplemental Nutrition Assistance Program:"

AMERICAN JOURNAL OF

Preventive Medicine

tR-EFREPORj

Improving the Nutritional Impact of the Supplemental

CrossMark

Nutrition Assistance Program: Perspectives From the Participants

Cindy W. Leung, ScD, MPH,1 Aviva A. Musicus, BA,2 Walter C. Willett, MD, DrPH,2,3,4

Eric B. Rimm, ScD2,3,4

Introduction: The Supplemental Nutrition Assistance Program (SNAP) is the largest federal food assistance program designed to alleviate food insecurity and improve dietary intake. This study assessed the opinions of SNAP participants and food-insufficient nonparticipants on their perceptions of the program and strategies to improve its nutritional impact.

Methods: This study surveyed 387 individuals via Amazon Mechanical Turk, of whom 118 were SNAP participants and 269 were food insufficient but not enrolled in SNAP (nonparticipants). Open-ended questions were coded and analyzed for thematic content. For closed-ended questions, response frequencies were compared using chi-square tests. Data were analyzed in 2016.

Results: SNAP participants reported that the program successfully served its primary purpose: to allow individuals to buy enough food to make ends meet and reduce food insecurity. Importance was placed on buying food for their children/families and the ability to allocate money for other expenses. To improve the nutritional impact, SNAP participants suggested more nutrition education, increasing the benefit allotment, incentivizing healthful foods, and excluding unhealthful foods for purchase with SNAP. When participants and nonparticipants were asked to choose between SNAP and a nutritionally enhanced program combining healthy incentives with exclusions for sugary beverages (i.e., SNAP+), 68% of participants and 83% of nonparticipants chose SNAP+. Of those who initially chose SNAP, 68% of participants and 64% of nonparticipants chose SNAP+ if paired with a 50% increase in total benefits.

Conclusions: SNAP participants and food-insufficient nonparticipants support policies that facilitate purchases of healthful foods and limit purchases of unhealthful foods, specifically sugary beverages. Am J Prev Med 2017;52(2S2):S193-S198. © 2016 American Journal of Preventive Medicine. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

INTRODUCTION

The Supplemental Nutrition Assistance Program (SNAP) is the largest federal food assistance program designed to alleviate food insecurity and improve dietary intake. A 2015 White House report highlighted the role of SNAP in lifting families out of poverty and hunger.1 However, the role SNAP plays in improving dietary intake is less clear. In recent years, various stakeholder groups have been engaged in a discussion about maximizing the nutritional impact of SNAP. However, SNAP participants and low-income nonparticipants have been missing from these conversations, despite their unique perspectives. The study assessed the opinions of SNAP participants and non-participants on: (1) perceptions of SNAP; (2) support for

policies to strengthen its positive nutritional impact; and (3) their preferences for an alternative program that

From the 'Center for Health and Community, University of California, San Francisco, San Francisco, California; 2Department of Nutrition, Harvard T. H. Chan School of Public Health, Boston, Massachusetts; 3Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts; and 4Channing Division of Network Medicine, Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts

Address correspondence to: Cindy W. Leung, ScD, MPH, Center for Health and Community, University of California, San Francisco, 3333 California Street, Suite 465, San Francisco CA 94118. E-mail: cindyleung@ post.harvard.edu.

This article is part of a supplement issue titled The Supplemental Nutrition Assistance Program's Role in Addressing Nutrition-Related Health Issues.

0749-3797/$36.00

http://dx.doi.org/10.1016Aj.amepre.2016.07.024

© 2016 American Journal of Preventive Medicine. Published by Elsevier Inc. Am J Prev Med 2017;52(2S2):S193-S198 S193

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

paired incentives for healthful food with exclusions of sugary beverages.

METHODS

Study Sample and Design

Respondents were recruited using Amazon Mechanical Turk (MTurk), a marketplace where individuals complete paid tasks for various organizations.2 Details and strengths of MTurk, including the ability to study low-income individuals and other hard-to-reach populations, have been described elsewhere.2-6 For this study, a task was posted describing "a short research survey about your food shopping habits." After obtaining consent, respondents were directed to Qualtrics to answer screening questions. The survey was restricted to adults aged >18 years, U.S. residents, and either receipt of SNAP benefits or an affirmative response to the U.S. Department of Agriculture food insufficiency screener in the past 12 months.7 Of the 743 total responses, 387 individuals met these eligibility criteria, of whom 118 were SNAP participants and 269 were nonparticipants (i.e., reported household food insufficiency but were not enrolled in SNAP in the past 12 months). Nonparticipants were included to examine the attitudes of individuals who previously applied for SNAP, received SNAP benefits, or may be likely to participate in SNAP in the future. However, given the nature of the survey, data were not collected on whether nonparticipants were truly eligible for SNAP participation. Safeguards were included so only one response

was recorded from each IP (Internet Protocol) address. Data were collected in 2015. The study was considered exempt by the University of California, San Francisco Committee on Human Research.

Respondents completed a 37-item survey about their demographics, grocery shopping habits, and the six-item short-form household food insecurity module.8 SNAP participants were asked 12 additional questions, including open-ended questions pertaining to perceived program strengths, strategies for overall improvement, and strategies to improve nutritional intake. Non-participants were asked ten questions about their perceptions of the program. Most closed-ended questions used identical wording as previous telephone surveys (Appendix, available online).9,10

Statistical Analysis

This study focused on the responses to the SNAP-specific questions. Using an inductive approach and the Framework Method, open-ended questions were analyzed for thematic content by two members of the research team (CWL and AAM).11 Briefly, a coding scheme of emergent themes and subthemes was developed and modified for each question until data saturation was reached. Responses were then independently reviewed, coded, and compared until consensus was achieved. For close-ended questions, response frequencies and proportions are reported. Variation by SNAP participation was examined using chi-square tests. Statistical analyses were performed using Stata/IC, version 12.0. Data were analyzed in 2016.

Table 1. Characteristics of Survey Respondents

Overall SNAP participants Nonparticipants

Characteristics (n=387) (n=118) (n=269) p-value

Age, years, M (SD) 36.9 (12.4) 38.4 (13.0) 36.3 (12.1) 0.12

Female sex 265 (68.5) 90 (76.3) 175 (65.1) 0.03

Race/ethnicity 0.53

White/Caucasian 308 (79.6) 93 (78.8) 215 (79.9)

African American 31 (8.0) 12 (10.2) 19 (7.1)

Othera 14 (3.6) 13 (11.0) 35 (13.0)

Children in household 0.08

None 201 (51.9) 56 (47.5) 145 (53.9)

One 84 (21.7) 26 (22.0) 58 (21.6)

Two 65 (16.8) 18 (15.3) 47 (17.5)

Three or more 37 (9.6) 18 (15.3) 19 (7.1)

Household income <0.0001

<$35,000 161 (41.6) 79 (67.0) 82 (30.5)

$35,000 to <$59,000 123 (31.8) 26 (22.0) 97 (36.1)

$59,000 to <$83,000 61 (15.8) 8 (6.8) 53 (19.7)

$83,000 to <$107,000 26 (6.7) 3 (2.5) 23 (8.6)

>$107,000 16 (4.1) 2 (1.7) 14 (5.2)

Household food securityb 0.22

High/marginal food security 112 (28.9) 34 (28.8) 78 (29.0)

Low food security 123 (31.8) 31 (26.3) 92 (34.2)

Very low food security 152 (39.3) 53 (44.9) 99 (36.8)

WIC participation in past 12 months 33 (8.5) 21 (17.8) 12 (4.5) < 0.0001

Note: Data are n (%) unless otherwise noted. Boldface Indicates statistical significance (p<0.05).

aCategory was combined due to small cell sizes. Other group includes individuals identifying as Hispanic, Asian, and multiracial. bFood insecurity categories were assigned according to the U.S. Department of Agriculture: 0-1 affirmative responses, high or marginal food security; 2-4 affirmative responses, low food security; 5-6 affirmative responses, very low food security.

SNAP, Supplemental Nutrition Assistance Program; WIC, Special Supplemental Nutrition Program for Women, Infants, and Children.

Leung et al / Am J Prev Med 2017;52(2S2):S193-S198 Table 2. Strengths of SNAP and Strategies for Improvement From the Perspectives of SNAP Participants

Theme Quote

Existing strengths of SNAP

Theme: SNAP helps me buy enough food to make ends meet "It's nice to have something to fall back on when we need food."

Subtheme: SNAP is a lifesaver "[SNAP] keeps people from starving to death."

Theme: The EBT card is easy to use "Food stamps are easy to use and are accepted at pretty much all grocery stores."

Subtheme: The EBT card is discreet "The card makes it less embarrassing."

Theme: SNAP benefits are dependable each month Strategies that can help SNAP further reduce hunger Theme: Shorten the application and renewal processes "The money is loaded on the card on the same day every month. I know it's there for me." "They need a better system for getting people approved. They need to make the process of getting approved faster."

Subtheme: Lower the eligibility criteria "Change [the] income guidelines. Make it easier for the needy to get them."

Subtheme: Improve customer service from EBT caseworkers "Have kinder, more understanding, less judgmental employees."

Theme: Increase the benefit allotment "They need to increase the benefits so that it covers the whole month."

Subtheme: Include benefits or cash allotments "Allow a small portion to be used for nonfood necessities, like feminine hygiene for nonfood necessities products, soap, toothpaste, etc." Strategies that can improve the nutritional impact of SNAP Theme: Increase the benefit allotment "Give reasonable amounts so we do not have to choose to eat unhealthy food that costs less and lasts longer."

Theme: Incentivize purchases of healthy foods "I would definitely give people more benefits if they only bought healthy foods, because it would give incentives to eat healthier."

Theme: Remove sugary beverages and/or junk food from foods allowed for purchase "Cut out unnecessary and unhealthy sodas and cut down on sugary, unhealthy foods."

Theme: Provide nutrition education and meal planning "Offer a program for families to attend so they can learn how to eat healthy, but also how to shop smarter and maximize their benefits."

EBT, Electronic Benefit Transfer; SNAP, Supplemental Nutrition Assistance Program.

RESULTS

Respondents came from 47 states across the U.S. The mean age was 36.9 (SD=12.4) years. Sixty-eight percent were women, 80% identified as white, and 48% lived in a household with children (Table 1). Among nonpartici-pants, 9.7% reported having applied for SNAP in the past year. There were no differences by SNAP participation with respect to age, race/ethnicity, children in the household, or food insecurity. However, SNAP participants were more likely to be female, have a lower household income, and report participation in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) in the past year.

Table 2 highlights the program strengths and strategies for improvement, identified by SNAP participants. The primary strength was the ability to buy enough food to make ends meet. This was often paired with broader psychosocial benefits, including less stress when purchasing food and feelings of support when no other funds were available. Respondents also discussed the importance of buying food for their children/families and the ability to allocate non-SNAP income for nonfood expenses.

To further reduce hunger, the primary improvement was to increase the benefit allotment, particularly for children, the elderly, and other vulnerable groups. Many SNAP participants also expressed the desire to buy household supplies (e.g., hygiene and paper products) with SNAP benefits. To improve dietary intake, SNAP participants suggested providing more nutrition education and meal planning tips. Other suggestions included increasing the benefit allotment, incen-tivizing healthful foods, and excluding sugary beverages and other unhealthful foods from SNAP.

Table 3 describes support for policies and programs to improve the nutritional impact of SNAP. Most SNAP participants (76%) and nonparticipants (81%) supported pairing monetary incentives for fruits and vegetables with exclusions for sugary beverages. Participants were asked to indicate their preference for: (1) SNAP+, a program combining healthy incentives and exclusions for sugary beverages; or (2) SNAP in its current form. For the same level of benefits, 68% of SNAP participants and 83% of nonparticipants chose SNAP+. Of those who initially chose SNAP in its current form, 68% of SNAP participants and 64% of nonparticipants chose SNAP+ if paired with a 50% increase in benefit level.

Table 3. Support for Policies and Programs to Improve the Nutritional Impact of SNAP

Variable SNAP participants, n (%) (n=118) Nonparticipants, n (%) (n=269) p-valuea

Providing additional money for fruits, vegetables, or other healthful foods 0.25

Strongly support 78 (66.1) 157 (58.4)

Support somewhat 26 (22.0) 82 (30.5)

Oppose somewhat 7 (5.9) 20 (7.4)

Strongly oppose 7 (5.9) 10 (3.7)

Removing sugary drinks from products allowed under SNAP <0.0001

Strongly support 38 (32.2) 141 (52.4)

Support somewhat 26 (22.0) 74 (27.5)

Oppose somewhat 36 (30.5) 43 (16.0)

Strongly oppose 18 (15.3) 11 (4.1)

Both providing additional money for healthful foods and removing sugary drinks from products allowed under SNAP 0.20

Strongly support 54 (45.8) 142 (52.8)

Support somewhat 36 (30.5) 75 (27.9)

Oppose somewhat 16 (13.6) 39 (14.5)

Strongly oppose 12 (10.2) 13 (4.8)

Providing SNAP participants more benefits to guarantee enough to eat and good nutrition 0.0003

Strongly support 84 (71.2) 144 (53.5)

Support somewhat 21 (17.8) 81 (30.1)

Oppose somewhat 12 (10.2) 28 (10.4)

Strongly oppose 1 (0.9) 16 (6.0)

Providing more nutrition education or cooking classes 0.04

Strongly support 65 (55.1) 142 (52.8)

Support somewhat 30 (25.4) 99 (36.9)

Oppose somewhat 18 (15.3) 21 (7.8)

Strongly oppose 5 (4.2) 7 (2.6)

Preference for an alternate program combining healthful food incentives and exclusions for sugary drinks (i.e., SNAP+)

At the current benefit level 0.001

SNAP+ 80 (67.8) 224 (83.3)

SNAP as it currently is 38 (32.2) 45 (16.7)

With 10% more benefits < 0.0001

SNAP+ 79 (67.0) 237 (88.1)

SNAP as it currently is 39 (33.1) 32 (11.9)

With 25% more benefits 0.001

SNAP+ 92 (78.0) 244 (90.7)

SNAP as it currently is 26 (22.0) 25 (9.3)

With 50% more benefits 0.31

SNAP+ 104 (88.1) 246 (91.5)

SNAP as it currently is 14 (11.9) 23 (8.6)

Note: Boldface indicates statistical significance (p<0.05).

ap-value based on y2 tests.

SNAP, Supplemental Nutrition Assistance Program.

DISCUSSION

Both SNAP participants and food-insufficient nonparticipants need representation in the discussion of reforming SNAP policies. Most stakeholder groups, including those surveyed here, are supportive of increasing the benefit allotment, increasing nutrition education, and

incentivizing healthful foods.12,13 The Healthy Incentives Pilot demonstrated that providing incentives for fruits and vegetables positively influences their consumption among SNAP participants.14 However, exclusions for sugary beverages have generated controversy. Those opposed have argued that exclusions are paternalistic

and unfairly limit the choices of SNAP participants, whereas those in favor reason that sugary beverages have no nutritional value and exacerbate health disparities in low-income populations.15-19 The results of this and other studies have shown that most SNAP participants support removing sugary beverages from the program, particularly if paired with incentives for healthful food.9,10,20,21

This is also the first study to examine whether program participants would select a nutritionally enhanced program over the status quo. For the same level of benefits, twice as many SNAP participants and five times as many nonparticipants selected SNAP+. Nonparticipants were more likely to select SNAP+ at every benefit level, which may be due to differences in income, gender, attitudes toward paternalism, or other unmeasured factors. Even so, the majority of SNAP participants selected SNAP+, which suggests SNAP participants generally desire a program that promotes healthful eating, even if their level of support is lower than that of nonparticipants.

Limitations

Although studies have shown that data obtained from MTurk are reliable,22-24 selection bias remains an issue. Compared with the general SNAP and low-income populations, the study sample was weighted toward women and whites, and the respondents are likely more technologically savvy, which may favor the proposed policies. However, the proportion of SNAP participants supporting exclusions for sugary beverages in this study was nearly identical to a prior national survey.10 Assessments of the opinions of SNAP participants are needed in the scientific literature; thus, more in-depth studies with balanced samples of SNAP participants and nonparticipants will help to improve understanding of their unique program perspectives.

CONCLUSIONS

The opinions of SNAP participants and nonparticipants with the potential for future SNAP participation are needed for informing future SNAP policies. Because both groups support improvements to the nutritional quality of offered foods, these policies may help to alleviate diet-related disparities in this vulnerable population.

ACKNOWLEDGMENTS

Publication of this article was supported by the Physicians Committee for Responsible Medicine. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Physicians Committee for Responsible Medicine or NIH.

CW Leung was supported by grant 1K99HD84758 from NIH.

No financial disclosures were reported by the authors of this paper.

SUPPLEMENTAL MATERIAL

Supplemental materials associated with this article can be found in the online version at http://dx.doi.org/10.1016/j. amepre.2016.07.024.

REFERENCES

1. The Council of Economic Advisers. Long-Term Benefits of the Supplemental Nutrition Assistance Program. Washington, DC: Executive Office of the President of the United States; 2015.

2. Mason W, Suri S. Conducting behavioral research on Amazon's Mechanical Turk. Behav Res Methods. 2012;44(1):1-23. http://dx.doi. org/10.3758/s13428-011-0124-6.

3. Smith NA, Sabat IE, Martinez LR, Weaver K, Xu S. A Convenient Solution: Using MTurk to Sample from Hard-To-Reach Populations. Ind Org Psychol. 2015;8(2):220-228. http://dx.doi.org/10.1017/iop. 2015.29.

4. Casler K, Bickel L, Hackett E. Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Comput Human Behav. 2013;29 (6):2156-2160. http://dx.doi.org/10.1016/j.chb.2013.05.009.

5. Hauser DJ, Schwarz N. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behav Res Methods. 2016;48(1):400-407. http://dx.doi.org/10.3758/ s13428-015-0578-z.

6. Rouse SV. A reliablity analysis of Mechanical Turk data. Comput Human Behav. 2015;43:304-307. http://dx.doi.org/10.1016Zj.chb.2014. 11.004.

7. U.S. household food security survey module: three-stage design, with screeners. 2012. www.ers.usda.gov/datafiles/Food_Security_in_the_ United_States/Food_Security_Survey_Modules/hh2012.pdf. Accessed October 9, 2015.

8. Blumberg SJ, Bialostosky K, Hamilton WL, Briefel RR. The effectiveness of a short form of the Household Food Security Scale. Am J Public Health. 1999;89(8):1231-1234. http://dx.doi.org/10.2105/AJPH.89.8.1231.

9. Leung CW, Ryan-Ibarra S, Linares A, et al. Support for policies to improve the nutritional impact of the Supplemental Nutrition Assistance Program in California. Am J Public Health. 2015;105(8):1576-1580. http://dx.doi.org/10.2105/AJPH.2015.302672.

10. Long MW, Leung CW, Cheung LW, Blumenthal SJ, Willett WC. Public support for policies to improve the nutritional impact of the Supplemental Nutrition Assistance Program (SNAP). Public Health Nutr. 2014;17(1):219-224. http://dx.doi.org/10.1017/S136898001200506X.

11. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117. http://dx.doi.org/10.1186/ 1471-2288-13-117.

12. Blumenthal SJ, Hoffnagle EE, Leung CW, et al. Strategies to improve the dietary quality of Supplemental Nutrition Assistance Program (SNAP) beneficiaries: an assessment of stakeholder opinions. Public Health Nutr. 2014;17(12):2824-2833. http://dx.doi.org/10.1017/ S1368980013002942.

13. Leung CW, Hoffnagle EE, Lindsay AC, et al. A qualitative study of diverse experts' views about barriers and strategies to improve the diets and health of Supplemental Nutrition Assistance Program (SNAP) Beneficiaries. J Acad Nutr Diet. 2013;113(1):70-76. http://dx.doi.org/ 10.1016/j.jand.2012.09.018.

14. Bartlett S, Klerman J, Olsho L, et al. Evaluation of the Healthy Incentives Pilot (HIP): Final Report. Alexandria, VA: Food and Nutrition Service, U.S. Department of Agriculture; 2014.

15. Barnhill A. Impact and ethics of excluding sweetened beverages from the SNAP program. Am J Public Health. 2011;101(11):2037-2043 http://dx.doi.org/10.2105/AJPH.2011.300225.

16. Ludwig DS, Blumenthal SJ, Willett WC. Opportunities to reduce childhood hunger and obesity: restructuring the Supplemental Nutrition Assistance Program (the Food Stamp Program). JAMA. 2012;308 (24):2567-2568. http://dx.doi.org/10.1001/jama.2012.45420.

17. American Heart Association, American Stroke Association. Facts: Decreasing Sugar-Sweetened Beverage Consumption: Policy Approaches to Address Obesity. Washington, DC: American Heart Association; 2015.

18. Gundersen C, Ziliak JP. Food insecurity and health outcomes. Health Aff (Millwood). 2015;34(11):1830-1839. http://dx.doi.org/10.1377/ hlthaff.2015.0645.

19. Laraia BA. Carrots, sticks, or carrot sticks?: using federal food policy to engineer dietary change. Am J Prev Med. 2012;43(4):456-457 http://dx.doi.org/10.1016Aj.amepre.2012.07.018.

20. Leung CW, Cluggish S, Villamor E, Catalano PJ, Willett WC, Rimm EB. Few changes in food security and dietary intake from

short-term participation in the Supplemental Nutrition Assistance Program among low-income Massachusetts adults. J Nutr Educ Behav. 2014;46(1):68-74. http://dx.doi.org/10.1016/j.jneb.2013.10.001.

21. National Commission on Hunger. Freedom from Hunger: An Achievable Goal for the United States of America. Washington, DC: National Commission on Hunger; 2015.

22. Bartneck C, Duenser A, Moltchanova E, Zawieska K. Comparing the similarity of responses received from studies in Amazon's Mechanical Turk to studies conducted online and with direct recruitment. PLoS One. 2015;10(4):e0121595. http://dx.doi.org/10.1371/journal. pone.0121595.

23. Buhrmester M, Kwang T, Gosling SD. Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect Psychol Sci. 2011;6(1):3-5. http://dx.doi.org/10.1177/1745691610393980.

24. Simons DJ, Chabris CF. Common (mis)beliefs about memory: a replication and comparison of telephone and Mechanical Turk survey methods. PLoS One. 2012;7(12):e51876. http://dx.doi.org/ 10.1371/journal.pone.0051876.