•  
  •  
 

Keywords

belongingness, international students, random sampling, response rate, salience, survey administration, survey fatigue

Abstract

Researchers and educators historically have administered surveys to collect data, track perceptions, and/or solicit opinions from undergraduates, including internationals, attending American institutions. To promote survey responses’ generalizability, the present report advocates the following recommendations for best practices in survey administration. 1) Use stratified random sampling methodology to identify which specific subset of students from the campus’ source population to invite to the survey. 2) Strive for, design, and implement an administration process that achieves a high response rate (preferably above 90%). 3) Choose an incentive that is truly salient for survey invitees to recruit and reward them for their time. 4) Dispatch only two reminder invitations to the randomly sampled undergraduates; and as a last resort for maximizing response rate, invite substitutes for those who decline the invitations. These recommendations were followed in an exploratory pilot study on international undergraduates’ belongingness that is reported and discussed here.

References

Alkan, N. (2016). Psychological sense of university membership: An adaptation study of the PSSM scale for Turkish university students, The Journal of Psychology, 150(4), 431–449. https://doi.org/10.1080/00223980.2015.1087373

Bennett, L., & Nair, C. S. (2010). A recipe for effective participation rates for web-based surveys. Assessment & Evaluation in Higher Education, 35(4), 357–365. https://doi.org/10.1080/02602930802687752

Bethlehem, J. (2010). Selection Bias in web surveys. International Statistical Review, 78(2), 161–188. https://doi.org/10.1111/j.1751-5823.2010.00112.x

Bethlehem, J., & Bakker, B. (2014). The impact of nonresponse on survey quality. Statistical Journal of the IAOS, 30, 243–248. https://doi.org/ 10.3233/SJI-140831

Bobbitt, Z. (December 28, 2020). Sampling with replacement vs. without replacement. https://www.statology.org/sampling-with-vs-without-replacement/

Brown, R. F., St. John, A., Hu, Y., & Sandhu, G. (2024). Differential electronic survey response: Does survey fatigue affect everyone equally? Journal of Surgical Research, 294, 191–197. https://doi.org/10.1016/j.jss.2023.09.072

Cantor, D., O’Hare, B., & O’Connor, K. (2008). The use of monetary incentives to reduce non-response in random digit dial telephone surveys. In J. M. Lepkowski, C. Tucker, J. M. Brick, E. de Leeuw, L. Japec, P. J. Lavrakas, M. W. Link, & R. L. Sangster (Eds.), Advances in telephone survey methodology (pp. 471–498). Wiley. https://doi.org/10.1002/9780470173404.ch22

Center for Postsecondary Research. (2024). NSSE 2024 administration overview. https://nsse.indiana.edu/nsse/reports-data/nsse-overview.html#irr

Chapman, D. W. (2003). To substitute or not to substitute—that is the question. The Survey Statistician, 48, 32–34. https://isi-iass.org/home/wp-content/uploads/N48.pdf

Chen, P-S. D. (2011). Finding quality responses: The problem of low-quality survey responses and its impact on accountability measures. Research in Higher Education, 52(7), 659–674. https://doi.org/10.1007/s11162-011-9217-4

Cochran, W. G. (1977). Sampling techniques (3rd ed.). Wiley.

Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60(6), 821–836. https://doi.org/10.1177/00131640021970934

Darmody, M., Groarke, S., & Mihut, G. (2022). Engagement of international students at Irish higher education institutions. Journal of International Students, 12(2), 795–816. https://doi.org/10.32674/jis.v12i4.3608

de Koning, R., Egiz, A., Kotecha, J., Ciuculete, A. C., Ooi, S. Z. Y., Bankole, N. D. A., Erhabor, J., Higginbotham, G., Khan, M., Dalle, D. U., Sichimba, D., Bandyopadhyay, S., & Kanmounye, U. S. (2021). Survey fatigue during the COVID-19 pandemic: An analysis of neurosurgery survey response rates. Frontiers in Surgery, 8(690680), 1–7. https://doi.org/10.3389/fsurg.2021.690680

DePountis, V. M., Pogrund, R. L., Griffin-Shirley, N., & Lan, W. Y. (2015). Technologies used in the study of advanced mathematics by students who are visually impaired in classrooms: Teachers’ perspectives. International Journal of Special Education, 77(2), 131–144. https://doi.org/10.1177/0145482X1510900403

Dillman, D. A. (1991). The design and administration of mail surveys. Annual Review of Sociology, 17, 225–249. https://www.jstor.org/stable/2083342

Fan, W., & Yan, Z. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2), 132–139. https://doi.org/10.1016/j.chb.2009.10.015

Fass-Holmes, B. (2022). Survey fatigue—Literature search and analysis of implications for student affairs policies and practices. Journal of Interdisciplinary Studies in Education, 11(1), 56–73. https://www.ojed.org/jise/article/view/3262

Federal Committee on Statistical Methodology. (2020). A framework for data quality. https://nces.ed.gov/fcsm/pdf/FCSM.20.04_A_Framework_for_Data_Quality.pdf

Fincham, J. E. (2008). Response rates and responsiveness for surveys, standards, and the journal. American Journal of Pharmaceutical Education, 72(2), 43. https://doi.org/10.5688/aj720243

Fosnacht, K., Sarraf, S., Howe, E., & Peck, L. K. (2017). How important are high response rates for college surveys? The Review of Higher Education, 40(2), 245–265. https://doi.org/10.1353/rhe.2017.0003

Frankel, M. R., & Frankel, L. R. (1987). Fifty years of survey sampling in the United States. The Public Opinion Quarterly, 51(Part 2 Supplement), S127–S138. https://www.jstor.org/stable/2749193

Fricker, R. D., Jr. (2008). Sampling methods for web and e-mail surveys. In N. Fielding, R. M. Lee, & G. Blank (Eds.) The SAGE handbook of online research methods (pp. 195–216). SAGE. https://doi.org/10.4135/9780857020055.n11

Frohlich, M. T. (2002). Techniques for improving response rates in OM survey research. Journal of Operations Management, 20(1), 53–62. https://doi.org/10.1016/S0272-6963(02)00003-7

Gofton, K. (1999, April 29). Data firms react to survey fatigue; a recent mistake over lifestyle questionnaires has forced marketers to seriously re-evaluate their approach to customers. Marketing. http://bi.galegroup.com/global/article/GALE|A54658550/2942e26bbc00e0a2ba126fcd05de3b92

Goodenow, C. (1993). The psychological sense of school membership among adolescents—Scale development and educational correlates. Psychology in the Schools, 30, 79–90. https://doi.org/10.1002/1520-6807(199301)30:1<79::AID-PITS2310300113>3.0.CO;2-X

Gopalan, M., & Brady, S. T. (2020). College students’ sense of belonging: A national perspective. Educational Researcher, 49(2), 134–137. https://doi.org/10.3102/0013189X19897622

Groves, R. M. (2011). Three eras of survey research. The Public Opinion Quarterly, 75(5), 861–871. https://doi.org/10.1093/poq/nfr057

Harvey, L. (2003). Student feedback [1]. Quality in Higher Education, 9(1), 3–20. https://doi.org/10.1080/13538320308164

Heerwegh, D. (2005). Effects of personal salutations in e-mail invitations to participate in a web survey. Public Opinion Quarterly, 69(4), 588–598. https://doi.org/10.1093/poq/nfi053

Hill, A., Roberts, J., Ewings, P., & Gunnell, D. (1997). Non-response bias in a lifestyle survey. Journal of Public Health Medicine, 19(2), 203–207. https://doi.org/10.1093/oxfordjournals.pubmed.a024610

Holtom, B., Baruch, Y., Aguinis, H., & Ballinger, G. A. (2022). Survey response rates: Trends and a validity assessment framework. Human Relations, 75(8), 1560–1584. https://doi.org/10.1177/00187267211070769

Innerview Team (2024). Survey response rates: Definition, importance, and improvement strategies. https://innerview.co/blog/survey-response-rates-a-comprehensive-guide-for-2024

Kalton, G. (2021). Introduction to survey sampling. Sage Research Methods. https://doi.org/10.4135/9781071909812

Kaplowitz, M. D., Lupi, F., Couper, P. P., & Thorp, L. (2012). The effect of invitation design on web survey response rates. Social Science Computer Review, 30(3), 339–349. https://doi.org/10.1177/0894439311419084

Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305

Krsmanovic, M. (2021). The synthesis and future directions of empirical research on international students in the United States: The insights from one decade. Journal of International Students, 11(1), 1–23. https://doi.org/10.32674/jis.v11i1.1955

Lin, W-F., Hewitt, G. J., & Videras, J. (2017). “I’m meltiiiiiing...”: The decline of response rates and the impact of nonresponse bias on the results of national surveys at small colleges. New Directions for Institutional Research, 173, 51–62. https://doi.org/10.1002/ir.20212

Lipka, S. (2011). Want data? Ask students. Again and again. The Chronicle of Higher Education. https://www.chronicle.com/article/Want-Data-Ask-Students-Again/128537

Liu, M., Kuriakose, N., Cohen, J., & Cho, S. (2016). Impact of web survey invitation design on survey participation, respondents, and survey responses. Social Science Computer Review, 34(5), 631–644. https://doi.org/10.1177/0894439315605606

Lynn, P. (2004). The use of substitution in surveys. The Survey Statistician, 49, 14–16. https://isi-iass.org/home/wp-content/uploads/N49.pdf

Madans, J. H., Earp, M. S., Blumberg, S. J., Christopher, E. M., Thompson, K. J., Sivinski, R. G., Frenk, S. M., Fakhouri, T. H. I., & Piscopo, K. D. (2023). Best practices for nonresponse bias reporting. https://www.fcsm.gov/assets/files/docs/FCSM%20NRBA%20Report%20062623.pdf

Marangel, S., & Baik, C. (2022). international students’ suggestions for what universities can do to better support their mental wellbeing. Journal of International Students, 12(4), 921–942. https://doi.org/10.32674/jis.v12i4.3877

Maslow, A. H. (1954). Motivation and personality. Harper & Row.

McCarthy, J. S., & Beckler, D. (1999, October 28–31). An analysis of the relationship between survey burden and nonresponse: If we bother them more are they less cooperative? [Conference presentation]. International Conference on Survey Non-Response, 1999, Portland, OR, United States.

McNair, D. E. (2009). Review of “Assessment methods for student affairs.” NASPA Journal, 46(3), 544–550. https://doi.org/10.2202/1949-6605.5026

Mohamad, D., & Manning, K. D. (2024). What Does it Mean to ‘Belong’? A narrative literature review of ‘belongingness’ among international higher education students. Journal of International Students, 14(1), 21–37. https://doi.org/10.32674/jis.v15i1.5783

Morrow, E. (2024). Scamming higher ed: An analysis of phishing content and trends. Computers in Human Behavior, 158, 108274. https://doi.org/10.1016/j.chb.2024.108274

Murphy, M. C., Gopalan, M., Carter, E. R., Emerson, K. T. U., Bottoms, B. L., & Walton, G. M. (2020). A customized belonging intervention improves retention of socially disadvantaged students at a broad-access university. Science Advances, 6, 1–7. http://doi.org/10.1126/sciadv.aba4677

Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4), 558–625. https://doi.org/10.2307/2342192

Office of Management and Budget. (2006). Standards and guidelines for statistical surveys. https://unstats.un.org/unsd/dnss/docs-nqaf/USA_standards_stat_surveys.pdf

Olsen, R. (2008). Self-selection bias. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (p. 809). Sage. https://doi.org/10.4135/9781412963947.n526

Olson, C. A. (2014). Survey Burden, Response Rates, and the Tragedy of the Commons. Journal of Continuing Education in the Health Professions, 34(2), 93–95. https://doi.org/10.1002/chp.21238

Petrovčič, A., Petrič, G., & Manfreda, K. L. (2016). The effect of email invitation elements on response rate in a web survey within an online community. Computers in Human Behavior, 56, 320–329. http://dx.doi.org/10.1016/j.chb.2015.11.025

Porter, S. R. (2004). Raising response rates: What works? New Directions for Institutional Research, 121, 5–21. https://doi.org/10.1002/ir.97

Porter, S. R. (2005). Survey research policies: An emerging issue for higher education. New Directions in Institutional Research, 127, 5–15. https://doi.org/10.1002/ir.152

Porter, S. R., & Whitcomb, M. E. (2003). The impact of lottery incentives on student survey response rates. Research in Higher Education, 44(4), 389-407. http://www.jstor.org/stable/40197312

Porter, S. R., & Whitcomb, M. E. (2004). Understanding the effect of prizes on response rates. New Directions for Institutional Research, 121, 51–62. https://doi.org/10.1002/ir.100

Porter, S. R., & Whitcomb, M. E. (2005). Non-response in student surveys: The role of demographics, engagement and personality. Research in Higher Education, 46(2), 127–152. https://doi.org/10.1007/s11162-004-1597-2

Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. New Directions in Institutional Research, 121, 63–73. https://doi.org/10.1002/ir.101

Rao, J. N. K. (1966). On the comparison of sampling with and without replacement. Review of the International Statistical Institute, 34(2), 125–138. https://doi.org/10.2307/1401762

Scheaffer, R. L., Mendenhall, W., & Ott, L. (1986). Elementary survey sampling (3rd ed.). Duxbury Press.

Sharma, G. (2017). Pros and cons of different sampling techniques. International Journal of Applied Research, 3(7), 749–752. https://www.allresearchjournal.com/archives/2017/vol3issue7/PartK/3-7-69-542.pdf

Sharp, L. M., & Frankel, J. (1983). Respondent burden: A test of Some Common Assumptions. The Public Opinion Quarterly, 47(1), 36–53. https://doi.org/10.1086/268765

Sheehan, K. B. (2006). E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2), JCMC621. https://doi.org/10.1111/j.1083-6101.2001.tb00117.x

Simpson, J., Branagan, C., Lau Cheng Von, V., & Williams, L. (2024). Spamming our students? The use of email as a mass communication tool in higher education. Perspectives: Policy and Practice in Higher Education, 29(3), 152–159. https://doi.org/10.1080/13603108.2024.2440705

Singer, E., & Couper, M. P. (2008). Do incentives exert undue influence on survey participation? Experimental evidence. Journal of Empirical Research on Human Research Ethics, 3(3), 49–56. https://doi.org/10.1525/jer.2008.3.3.49

Singer, E., & Ye, C. (2013). The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 112–141. https://doi.org/10.1177/0002716212458082

Sinickas, A. (2007). Finding a cure for survey fatigue. Strategic Communication Management, 11(2). https://www.sinicom.com/wp-content/uploads/2018/03/article93.pdf

Steinmetz, C., Thompson, S., & Marshall, N. (2020). Surveying international university students: The case of the 5% response rate. Issues in Educational Research, 30(3), 1105–1125. https://www.iier.org.au/iier30/steinmetz.pdf

Stephan, F. F. (1948). History of the uses of modern sampling procedures. Journal of the American Statistical Association, 43(241), 12–39. https://www.jstor.org/stable/2280064

Sterba, S., & Foster, E. (2008). Self-selected Sample. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (pp. 807–808). Sage. https://doi.org/10.4135/9781412963947.n525

Stolzenberg, E. B., Aragon, M. C., Romo, E., Couch, V., McLennan, D., Eagan, M. K., & Kang, N. (2020). The American Freshman: National Norms Fall 2019. Higher Education Research Institute, UCLA. https://www.heri.ucla.edu/monographs/TheAmericanFreshman2019-Expanded.pdf

Stone, A. A., Schneider, S., Smyth, J. M., Junghaenel, D. U., Couper, M. P., Wen, C., Mendez, M., Velasco, S., & Goldstein, S. (2024). A population-based investigation of participation rate and self-selection bias in momentary data capture and survey studies. Current Psychology, 43(3), 2074–2090. https://doi.org/10.1007/s12144-023-04426-2

Strayhorn, T. L. (2012). College students’ sense of belonging: A key to educational success for all students (1st ed.). Routledge. https://doi.org/10.4324/9780203118924

Thach, L. (1995). Using electronic mail to conduct survey research. Educational Technology, 35(2), 27–31. https://www.jstor.org/stable/44428960

Trespalacios, J. H., & Perkins, R. A. (2016). Effects of personalization and invitation email length on web-based survey response rates. Tech Trends, 60, 330–335. https://doi.org.10.1007/s11528-016-0058-z

Tschepikow, W. K. (2012). Why don’t our students respond? Understanding declining participation in survey research among college students. Journal of Student Affairs Research and Practice, 49(4), 447–462. https://doi.org/10.1515/jsarp-2012-6333

University of California. (2024). Student survey completion and response rates. https://www.universityofcalifornia.edu/about-us/information-center/student-survey-completion-and-response-rates

U.S. Citizenship and Immigration Services. (February 21, 2023). Common scams. https://www.uscis.gov/scams-fraud-and-misconduct/avoid-scams/common-scams

Van Mol, C. (2017). Improving web survey efficiency: The impact of an extra reminder and reminder content on web survey response. International Journal of Social Research Methodology: Theory & Practice, 20(4), 317–327. https://doi.org/10.1080/13645579.2016.1185255

Vinson, M. N. (1996). The pros and cons of 360-degree feedback: Making it work. Training & Development, 50(4), 11–12. https://www.td.org/magazines/td-archive/1996/the-pros-and-cos-of-360degree-feedback-making-it-work

Wagle, R., Dowdy, E., Yang, C., Palikara, O., Castro, S., Nylund-Gibson, K., & Furlong, M. J. (2018). Preliminary investigation of the psychological sense of school membership scale with primary school students in a cross-cultural context. School Psychology International, 39(6), 568–586. https://doi.org/10.1177/0143034318803670

Walton, G. M., Murphy, M. C., Logel, C., Yeager, D. S., Goyer, J. P., Brady, S. T., Emerson, K. T. U., Paunesku, D., Fotuhi, O., Blodorn, A., Boucher K. L., Carter, E. R., Gopalan, M., Henderson, A., Kroeper K. M., Murdock-Perriera, L. A., Reeves, S. L., Ablorh T. T., Ansari, S.,…Krol, N. (2023). Where and with whom does a brief social-belonging intervention promote progress in college? Science, 380(6644), 499–505. http://doi.org/10.1126/science.ade4420

Wise, V. L., & Barham, M. A. (2012). Assessment matters: Moving beyond surveys. About Campus, 17(2), 26–29. https://doi.org/10.1002/abc.21077

You, S., Ritchey, K. M., Furlong, M. J., Shochet, I., & Boman, P. (2011). Examination of the Latent Structure of the Psychological Sense of School Membership Scale. Journal of Psychoeducational Assessment, 29(3), 225–237. https://doi.org/10.1177/0734282910379968

Young, A. M., Wendel, P. J., Esson, J. M., & Plank, K. M. (2018). Motivational decline and recovery in higher education STEM courses. International Journal of Science Education, 40(9), 1016–1033. https://doi.org/10.1080/09500693.2018.1460773

Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., & Aldridge, J. W. (2009). A Neural Computational Model of Incentive Salience. PLoS Computational Biology, 5(7), 1–14. https://doi.org/10.1371/journal.pcbi.1000437

Download

Share

COinS
 
 

To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.