Online Social Media Analytics Software as a Tool for Automating Data Collection: Concurrent Validity and Feasibility Study
Keywords:
Facebook, Social Media Analyzer, Concurrence Validity, Concordance Correlation Coefficient, Intraclass Correlation Coefficient, FeasibilityAbstract
Background: Facebook based research is emerging and social media analytics software may be a tool that could lower the cost of research. Unfortunately, the quality of data it extracts has not be documented or validated.
Objectives: To test accessibility and efficiency of social media analytics software in a Facebook based study and to test concurrent validity of Likes extracted.
Methods: We conducted a review of accessible online social media analytics software and selected one for a case study comparing it to manual extraction procedures. Thereafter we tested concurrent validity of the social media analyzer as a method for extracting Likes from Facebook pages. The agreement in Likes extracted was tested with intraclass correlation coefficient (ICC), concordance correlation coefficient (CCC), and Bland and Altman plot.
Results: Eighteen software were found with five being completely free. The selected software was used in the completion of a case study at no cost but took a longer time to extract data compared to manual extraction procedures. Exact data points were matched in only a few pages (n=20, 33.9%) but differences between Likes extracted by the software and manual extraction was not statistically different (p=0.471). The software was found to have perfect ICC for half of the studies with the rest having “almost perfect” agreement (ICC = 0.97 and ICC = 0.98, for 3rd and 4th quartile, respectively). Concurrent validity was high (CCC = 0.995) with Bland and Altman plot showing only 5% of measurements outside 95% agreement level.
Conclusion: Social media analyzer software are accessible and can be used at no cost. Facebook Likes extracted through software compared to Likes manually extracted may not be exact matches but have strong agreement and validity.
References
Adel, R. (2019). List: certified senatorial candidates for 2019 elections. Retrieved from https://www.philstar.com/headlines/2019/02/06/1891374/list-certified-senatorial-candidates-2019-elections/amp/
Alemayehu C, Mitchell G, Nikles J. (2018). Barriers for conducting clinical trials in developing countries- a systematic review. Int J Equity Health. 17(1):37. doi:10.1186/s12939-018-0748-6
American Psychological Association. (2010). Ethical principles of psychologists and code of conduct. Available at: https://www.apa.org/ethics/code/principles.pdf (accessed 4 August 2016)
Aslam, S. (2019). Facebook by the numbers: stats, demographics, & fun facts. Retrieved from https://www.omnicoreagency.com/facebook-statistics/
Bland, J.M., Altman, D.J. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 8:307–310.
Buchanan, L., Yeatman, H., Kelly, B., Kariippanon, K. (2018), A thematic content analysis of how marketers promote energy drinks on digital platforms to young Australians. Australian and New Zealand Journal of Public Health, 42: 530-531. doi:10.1111/1753-6405.12840
Cole, A., Leonard, M.T., McAuslan, P. (2018) Social media and couples: what are the important factors for understanding relationship satisfaction? Cyberpsychology, Behavior, and Social Networking, 21:9. http://doi.org/10.1089/cyber.2017.0425
Eichstaedt, J.C., Smith, R.J., Merchant, R.M., Ungar, L.H., Crutchley, P., Preoţiuc-Pietro, D., Asch, D.A., Schwartz, H.A. (2018) Facebook language predicts depression in medical records. Proc Natl Acad Sci USA, 115(44): 11203–11208. doi:10.1073/pnas.1802331115
Eranti, V., Lonkila, M. (2015). The social significance of the Facebook Like button. First Monday, 20(6). doi: http://dx.doi.org/10.5210/fm.v20i6.5505
Eysenbach, G., Till, J.E. (2001). Ethical issues in qualitative research on Internet communities. BMJ. 323(7321):1103–1105.
Facebook (2019). Stats. Retrieved from https://newsroom.fb.com/company-info/
Facebook. Data Policy. Retrieved from https://www.facebook.com/full_data_use_policy
Fiks, A. G., Gruver, R. S., Bishop-Gilyard, C. T., Shults, J., Virudachalam, S., Suh, A. W., Gerdes, M., Kalra, G.K., DeRusso, P.A., Lieberman, A., Weng, D., Elovitz, M.A., Berkowitz, R.I., Power, T.J. (2017). A social media peer group for mothers to prevent obesity from infancy: the
Grow2Gether randomized trial. Childhood Obesity, 13(5): 356–368. doi:10.1089/chi.2017.0042
Flicker, S., Haans, D., Skinner, H. (2004). Ethical dilemmas in research on Internet communities. Qual Health Res. 14(1):124–134. doi: 10.1177/1049732303259842.
Gough, A., Hunter, R. F., Ajao, O., Jurek, A., McKeown, G., Hong, J., Barrett, E., Ferguson, M., McElwee, G., McCarthy, M., Kee, F. (2017). Tweet for Behavior Change: Using Social Media for the Dissemination of Public Health Messages. JMIR public health and surveillance, 3(1), e14. doi:10.2196/publichealth.6313
Hudson, J.M., Bruckman, A. (2004). ‘Go away’: Participant objections to being studied and the ethics of chatroom research. The Information Society. 20(2): 127–139.
Inkster, B., Stillwell, D., Kosinski, M., Jones, P. (2016). A decade into Facebook: where is psychiatry in the digital age? Lancet Psychiatry, 3 (11): 1087-1090. https://doi.org/10.1016/S2215-0366(16)30041-4
Kessler, A. (2012). The button that made Facebook billions. Retrieved from https://www.wsj.com/articles/SB10001424052970204652904577196992203069570
Kite, J., Foley, B.C., Grunseit, A.C., Freeman, B. (2016). Please Like me: Facebook and public health communication. PLoS ONE, 11(9): e0162765. doi:10.1371/journal.pone.0162765
Kosinski, M., Stillwell, D., Graepel, T. (2013) Private traits and attributes are predictable from digital records of human behavior. Proc Natl Acad Sci USA. 110(15):5802-5. doi: 10.1073/pnas.1218772110.
Kristensen, J.B., Albrechtsen, T., Dahl-Nielsen, E., Jensen, M., Skovrind, M., Bornakke. T. (2017). Parsimonious data: How a single Facebook like predicts voting behavior in multiparty systems. PLoS ONE, 12(9): e0184562. https://doi.org/ 10.1371/journal.pone.0184562
Lee, K. (2019). Know what's working on social media: 27 paid and free social media analytics tool. Retrieved from https://buffer.com/library/social-media-analytics-tools
Lin, L. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics. 45: 255–268.
Liu, J., Tang, W., Chen, G., Lu, Y., Feng, C., Tu, X.M. (2016). Correlation and agreement: overview and clarification of competing concepts and measures. Shanghai Arch Psychiatry. 28(2):115–120. doi:10.11919/j.issn.1002-0829.216045
McBride, G.B. (2005). A proposal for strength-of-agreement criteria for Lin's Concordance Correlation Coefficient. [PDF file] NIWA Client Report: HAM2005-062; p. 6. Retrieve from https://www.medcalc.org/download/pdf/McBride2005.pdf
Moreno, M. A., Kerr, B., Lowry, S. J. (2018). A longitudinal investigation of associations between marijuana displays on Facebook and self-reported behaviors among college students, Journal of Adolescent Health. doi:10.1016/j.jadohealth.2018.03.017
Park, B.K., Calamaro, C. (2013). A systematic review of social networking sites: Innovative platforms for health research targeting adolescents and young adults. J Nurs Scholarsh. 45(3):256–264. doi: 10.1111/jnu.12032.
Pearlman, L. (2009). “I like this”. Retrieved from https://www.facebook.com/notes/facebook/i-like-this/53024537130
Perrin, A., & Anderson, M. (2019). Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018. Retrieved from https://www.pewresearch.org/fact-tank/2019/04/10/share-of-u-s-adults-using-social-media-including-facebook-is-mostly-unchanged-since-2018/
Pittman, M. (2018). Happiness, loneliness, and social media: perceived intimacy mediates the emotional benefits of platform use. The Journal of Social Media in Society, 7 (2): 164-176.
Portney, L.G., Watkins, M.P. (2000). Foundations of clinical research Applications to practice. Prentice Hall Inc. New Jersey ISBN 0-8385-2695-0. 560-567.
R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/
Ringelhan, S., Wollersheim, J., Welpe, I.M. (2015). I Like, I cite? Do Facebook Likes predict the impact of scientific work? PLoS ONE, 10(8): e0134389. doi:10.1371/journal.pone.0134389
Samuel-Azran, T., Yarchi, M., Wolfsfeld, G. (2017). Engagement and likeability of negative messages on Facebook during Israel’s 2013 elections. The Journal of Social Media in Society, 6(1): 42-68.
Serugaa, B., Sadikovb, A., Cazapc, E.L., Delgadod, L.B., Digumartie, R., Leighlf, N.B., Meshrefg, M.M., Minamih, H., Robinsoni, E., Yamaguchij, N.H., Pylek, D., Cuferl, T. (2013). Barriers and challenges to global clinical cancer research. Oncologist. 19(1):61–67. doi:10.1634/theoncologist.2013-0290
Shrout, P.E, & Fleiss, J.L. (1979). Intraclass correlation: uses in assessing rater reliability. Physiological Bulletin. 86:420–428.
Starling, M.S., Kandel, Z., Haile, L., Simmons, R.G. (2018). User profile and preferences in fertility apps for preventing pregnancy: an exploratory pilot study. Mhealth, 4:21. doi:10.21037/mhealth.2018.06.02
Tefertiller, A. (2017). Like us on Facebook: social capital, opinion leadership, and social media word-of-mouth for promoting cultural goods.
The Journal of Social Media in Society, 7(2): 274-296.
Walker, M., King, G., Hartman, L. (2018). Exploring the potential of social media platforms as data collection methods for accessing and understanding experiences of youth with disabilities: a narrative review. The Journal of Social Media in Society, 7 (2): 43-68.
Whitaker, C., Stevelink, S., & Fear, N. (2017). The use of Facebook in recruiting participants for health research purposes: a systematic review. J Med Internet Res, 19(8): e290. Published 2017 Aug 28. doi:10.2196/jmir.7071
Wilkinson, D., Thelwall, M. (2011). Researching personal information on the public web: methods and ethics. Social Science Computer Review. 29(4): 387–401
Wilson, R.E., Gosling, S.D., & Graham, L.T. (2012). A review of Facebook research in the social sciences. Perspectives on Psychological Science, 7(3): 203-20.
Youyou, W., Kosinski, M., Stillwell, D. (2015) Computer-based personality judgments are more accurate than those made by humans. Proc Natl Acad Sci USA, 112: 1036–40.
Zhang, X. (2018). Social media popularity and election results: a study of the 2016 Taiwanese general election. PLos ONE. 13(11): e0208190. doi: 10.1371/journal.pone.0208190
Zimmer, M. (2010). “But the data is already public”: On the ethics of research in Facebook. Ethics Inf Technol. 12(4):313–325. doi: 10.1007/s10676-010-9227-5.
Alemayehu C, Mitchell G, Nikles J. (2018). Barriers for conducting clinical trials in developing countries- a systematic review. Int J Equity Health. 17(1):37. doi:10.1186/s12939-018-0748-6
American Psychological Association. (2010). Ethical principles of psychologists and code of conduct. Available at: https://www.apa.org/ethics/code/principles.pdf (accessed 4 August 2016)
Aslam, S. (2019). Facebook by the numbers: stats, demographics, & fun facts. Retrieved from https://www.omnicoreagency.com/facebook-statistics/
Bland, J.M., Altman, D.J. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 8:307–310.
Buchanan, L., Yeatman, H., Kelly, B., Kariippanon, K. (2018), A thematic content analysis of how marketers promote energy drinks on digital platforms to young Australians. Australian and New Zealand Journal of Public Health, 42: 530-531. doi:10.1111/1753-6405.12840
Cole, A., Leonard, M.T., McAuslan, P. (2018) Social media and couples: what are the important factors for understanding relationship satisfaction? Cyberpsychology, Behavior, and Social Networking, 21:9. http://doi.org/10.1089/cyber.2017.0425
Eichstaedt, J.C., Smith, R.J., Merchant, R.M., Ungar, L.H., Crutchley, P., Preoţiuc-Pietro, D., Asch, D.A., Schwartz, H.A. (2018) Facebook language predicts depression in medical records. Proc Natl Acad Sci USA, 115(44): 11203–11208. doi:10.1073/pnas.1802331115
Eranti, V., Lonkila, M. (2015). The social significance of the Facebook Like button. First Monday, 20(6). doi: http://dx.doi.org/10.5210/fm.v20i6.5505
Eysenbach, G., Till, J.E. (2001). Ethical issues in qualitative research on Internet communities. BMJ. 323(7321):1103–1105.
Facebook (2019). Stats. Retrieved from https://newsroom.fb.com/company-info/
Facebook. Data Policy. Retrieved from https://www.facebook.com/full_data_use_policy
Fiks, A. G., Gruver, R. S., Bishop-Gilyard, C. T., Shults, J., Virudachalam, S., Suh, A. W., Gerdes, M., Kalra, G.K., DeRusso, P.A., Lieberman, A., Weng, D., Elovitz, M.A., Berkowitz, R.I., Power, T.J. (2017). A social media peer group for mothers to prevent obesity from infancy: the
Grow2Gether randomized trial. Childhood Obesity, 13(5): 356–368. doi:10.1089/chi.2017.0042
Flicker, S., Haans, D., Skinner, H. (2004). Ethical dilemmas in research on Internet communities. Qual Health Res. 14(1):124–134. doi: 10.1177/1049732303259842.
Gough, A., Hunter, R. F., Ajao, O., Jurek, A., McKeown, G., Hong, J., Barrett, E., Ferguson, M., McElwee, G., McCarthy, M., Kee, F. (2017). Tweet for Behavior Change: Using Social Media for the Dissemination of Public Health Messages. JMIR public health and surveillance, 3(1), e14. doi:10.2196/publichealth.6313
Hudson, J.M., Bruckman, A. (2004). ‘Go away’: Participant objections to being studied and the ethics of chatroom research. The Information Society. 20(2): 127–139.
Inkster, B., Stillwell, D., Kosinski, M., Jones, P. (2016). A decade into Facebook: where is psychiatry in the digital age? Lancet Psychiatry, 3 (11): 1087-1090. https://doi.org/10.1016/S2215-0366(16)30041-4
Kessler, A. (2012). The button that made Facebook billions. Retrieved from https://www.wsj.com/articles/SB10001424052970204652904577196992203069570
Kite, J., Foley, B.C., Grunseit, A.C., Freeman, B. (2016). Please Like me: Facebook and public health communication. PLoS ONE, 11(9): e0162765. doi:10.1371/journal.pone.0162765
Kosinski, M., Stillwell, D., Graepel, T. (2013) Private traits and attributes are predictable from digital records of human behavior. Proc Natl Acad Sci USA. 110(15):5802-5. doi: 10.1073/pnas.1218772110.
Kristensen, J.B., Albrechtsen, T., Dahl-Nielsen, E., Jensen, M., Skovrind, M., Bornakke. T. (2017). Parsimonious data: How a single Facebook like predicts voting behavior in multiparty systems. PLoS ONE, 12(9): e0184562. https://doi.org/ 10.1371/journal.pone.0184562
Lee, K. (2019). Know what's working on social media: 27 paid and free social media analytics tool. Retrieved from https://buffer.com/library/social-media-analytics-tools
Lin, L. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics. 45: 255–268.
Liu, J., Tang, W., Chen, G., Lu, Y., Feng, C., Tu, X.M. (2016). Correlation and agreement: overview and clarification of competing concepts and measures. Shanghai Arch Psychiatry. 28(2):115–120. doi:10.11919/j.issn.1002-0829.216045
McBride, G.B. (2005). A proposal for strength-of-agreement criteria for Lin's Concordance Correlation Coefficient. [PDF file] NIWA Client Report: HAM2005-062; p. 6. Retrieve from https://www.medcalc.org/download/pdf/McBride2005.pdf
Moreno, M. A., Kerr, B., Lowry, S. J. (2018). A longitudinal investigation of associations between marijuana displays on Facebook and self-reported behaviors among college students, Journal of Adolescent Health. doi:10.1016/j.jadohealth.2018.03.017
Park, B.K., Calamaro, C. (2013). A systematic review of social networking sites: Innovative platforms for health research targeting adolescents and young adults. J Nurs Scholarsh. 45(3):256–264. doi: 10.1111/jnu.12032.
Pearlman, L. (2009). “I like this”. Retrieved from https://www.facebook.com/notes/facebook/i-like-this/53024537130
Perrin, A., & Anderson, M. (2019). Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018. Retrieved from https://www.pewresearch.org/fact-tank/2019/04/10/share-of-u-s-adults-using-social-media-including-facebook-is-mostly-unchanged-since-2018/
Pittman, M. (2018). Happiness, loneliness, and social media: perceived intimacy mediates the emotional benefits of platform use. The Journal of Social Media in Society, 7 (2): 164-176.
Portney, L.G., Watkins, M.P. (2000). Foundations of clinical research Applications to practice. Prentice Hall Inc. New Jersey ISBN 0-8385-2695-0. 560-567.
R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/
Ringelhan, S., Wollersheim, J., Welpe, I.M. (2015). I Like, I cite? Do Facebook Likes predict the impact of scientific work? PLoS ONE, 10(8): e0134389. doi:10.1371/journal.pone.0134389
Samuel-Azran, T., Yarchi, M., Wolfsfeld, G. (2017). Engagement and likeability of negative messages on Facebook during Israel’s 2013 elections. The Journal of Social Media in Society, 6(1): 42-68.
Serugaa, B., Sadikovb, A., Cazapc, E.L., Delgadod, L.B., Digumartie, R., Leighlf, N.B., Meshrefg, M.M., Minamih, H., Robinsoni, E., Yamaguchij, N.H., Pylek, D., Cuferl, T. (2013). Barriers and challenges to global clinical cancer research. Oncologist. 19(1):61–67. doi:10.1634/theoncologist.2013-0290
Shrout, P.E, & Fleiss, J.L. (1979). Intraclass correlation: uses in assessing rater reliability. Physiological Bulletin. 86:420–428.
Starling, M.S., Kandel, Z., Haile, L., Simmons, R.G. (2018). User profile and preferences in fertility apps for preventing pregnancy: an exploratory pilot study. Mhealth, 4:21. doi:10.21037/mhealth.2018.06.02
Tefertiller, A. (2017). Like us on Facebook: social capital, opinion leadership, and social media word-of-mouth for promoting cultural goods.
The Journal of Social Media in Society, 7(2): 274-296.
Walker, M., King, G., Hartman, L. (2018). Exploring the potential of social media platforms as data collection methods for accessing and understanding experiences of youth with disabilities: a narrative review. The Journal of Social Media in Society, 7 (2): 43-68.
Whitaker, C., Stevelink, S., & Fear, N. (2017). The use of Facebook in recruiting participants for health research purposes: a systematic review. J Med Internet Res, 19(8): e290. Published 2017 Aug 28. doi:10.2196/jmir.7071
Wilkinson, D., Thelwall, M. (2011). Researching personal information on the public web: methods and ethics. Social Science Computer Review. 29(4): 387–401
Wilson, R.E., Gosling, S.D., & Graham, L.T. (2012). A review of Facebook research in the social sciences. Perspectives on Psychological Science, 7(3): 203-20.
Youyou, W., Kosinski, M., Stillwell, D. (2015) Computer-based personality judgments are more accurate than those made by humans. Proc Natl Acad Sci USA, 112: 1036–40.
Zhang, X. (2018). Social media popularity and election results: a study of the 2016 Taiwanese general election. PLos ONE. 13(11): e0208190. doi: 10.1371/journal.pone.0208190
Zimmer, M. (2010). “But the data is already public”: On the ethics of research in Facebook. Ethics Inf Technol. 12(4):313–325. doi: 10.1007/s10676-010-9227-5.
Downloads
Published
2020-12-18
Issue
Section
Articles
License
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
