2008 SMART SURVEY DESIGN This guide provides information on writing successful and effective survey questions, creating survey flow and layout, calculating response rates, tips for increasing response rates, and the pros and cons of online surveys. (Plus an appendix of links and works cited for additional help in survey design.
Smart Survey Design Table of Contents & Summary I. Introduction: Relevancy and Accuracy…..………………………pg. 4 Be familiar with the question Know the objectives Know the kind of information needed Pay attention to wording style, type, and question sequence Pay attention to survey length and make the survey interesting Collect the data in a reliable and valid way to ensure accuracy A. Question Intent……………………………………………………pg.
Smart Survey Design B. Question Sequence…………………………………………. pg. 14 Opening questions Question flow Location of sensitive questions C. Survey Layout………………………………………………... pg. 15 1. Basic guidelines ♦ Introduction ♦ Body of the survey design ♦ End of survey or thank you page 2. Layout for coding/editing and identification ♦ Identification ♦ Numbering ♦ Instructions ♦ Fonts & formats D. Pre-Test………………………………………………………… pg.
Smart Survey Design ♦ Better at addressing sensitive issues ♦ Cost efficient ♦ Faster delivery ♦ Endless design options ♦ Dynamic ♦ Ability to track ♦ Quick response time ♦ Easier to use for complex routing or skip logic ♦ Automatic Randomization of Answer Choices 2. Cons: ♦ SPAM/Privacy ♦ Technical issues ♦ Submitting multiple submissions ♦ Not having an interviewer present to clarify questions IV. Collecting secure data and privacy best practices……... pg. 26 V. Section 508 & Accessibility ………………………………….
Smart Survey Design I: INTRODUCTION: RELEVANCY AND ACCURACY Writing solid and unbiased surveys may seem like a daunting and unapproachable task. However, it is not as overwhelming as one may think! Creating well structured, simply written questions will help in collecting valid survey responses. An important goal as a survey author is to construct clear, direct questions and answers using the language that survey participants will understand.
Smart Survey Design clear, direct, and brief questions will help the survey respondents to know exactly what you are asking. By making sure that the questions asked do not have more than one possible meaning also helps in preventing respondents‟ confusion. Asking sensitive questions in alternate ways may help to alleviate respondents‟ concerns. For example, many people may feel that income, age, lifestyle habits, etc. are personal and may not want to disclose that information.
Smart Survey Design respondents may get into a pattern of response that does not reflect their actual thoughts. They may simply just click the 1st rating scale every time to answer every question and finish the survey quickly (Brace 2004, 18). B. Constructing Good vs. Bad Question s: Each survey question has a unique need. Because of this, there is no universal right or wrong of “question wording.” However, there are ways to construct good vs. bad ones.
Smart Survey Design be accomplished with the “randomized answers” option as a Professional subscriber in SurveyMonkey. Example of Leading Question Bias: Example: We have recently upgraded SurveyMonkey‟s features to become a first class tool. What are your thoughts on the new site? Replace with: What are your thoughts on the upgrades to SurveyMonkey? B. Avoid loaded questions – This type of answer bias works through emotionally charged items like words, stereotypes, or prestige images.
Smart Survey Design Example of a Double-Barreled Question: Example: “How useful do you find SurveyMonkey‟s Help Center Topics and the email support center?” Replace with: Question 1: “How useful do you find SurveyMonkey‟s Help Center Topics?” Question 2: “How useful do you find SurveyMonkey‟s email support?” 4.)Be Specific – Ask precise questions. Avoid things that are too general, too complex or undefined. Stay away from using words like “often,” “usually,” “generally,” etc.
Smart Survey Design II: SURVEY QUESTION TYPES AND SURVEY STRUCTURE A. Question Types: The types of survey questions used in a survey will play a role in producing unbiased or relevant survey responses. As the survey designer, consider the types of questions to use and when it is appropriate to use them. Question types range from open-ended (comments to essays) to closed-ended (yes/no, multiple choice, rating scale, etc).
Smart Survey Design another type is the “mutli-chotomous” question, which allows respondents to choose one of many answer choices (Brace 2004, 55-67).
Smart Survey Design Example: Please rank the items from 1 to 5 according to what is most important when looking for an online survey provider. st 1 choice rd th 2nd choice 3 choice 4 choice th 5 choice Survey templates Multi-lingual Support More customization Multiple logins More export formats 3.
Smart Survey Design Replace with: Descriptions on Every Rating Scale: Please answer the following about SurveyMonkey. 1 – Strongly Disagree 2 - Disagree 3 – Neutral 4 – Agree 5 – Strongly Agree It is easy to use. It is fun. The two common types of matrix-rating scales are called likert and semantic differential scales. Rating scales are popular ways of collecting subjective data where you want to measure a respondent‟s ideas (e.g.
Smart Survey Design typically recommended to use a seven–point scale for these types. It is also good to keep the statements on the opposite ends short and precise (Brace 2004, 89-90). NOTE: Currently SurveyMonkey does not provide this format in our available question types. We are working on some feature requests and hope to have this offered in the future.
Smart Survey Design statements tend to be selected. If you want to avoid potential bias it is good to balance the scale (Brace 2004, 81). There are some occasions in which an unbalanced scale is suitable. For example, in a customer satisfaction survey, few customers may say that something is “unimportant.” In the example scale below, the “important” will become the midpoint.
Smart Survey Design 1. Opening questions – The first few questions in the survey should be easy and interesting in order to calm any participants‟ suspicions about the survey‟s integrity. This allows the participants to build up confidence in the survey‟s objective. In return, this may stimulate their interest and overall participation (Iraossi 2006, 74-78). 2. Question flow – The question sequence in the survey body should take on a flow of ideas and be geared towards the respondents‟ abilities.
Smart Survey Design help ease those concerns. You may also want to provide an estimate of how long the survey might take or whether you are offering any kind of incentive or prize for taking the survey. Remember to deliver on your promised gift! If you provide this information up front it usually leads to honest responses and more completed surveys.
Smart Survey Design b. Body of the Survey Design: The use of space throughout the survey is also important. Trying to fit too much information (e.g. too many questions) on a single page may cause respondents to struggle through the survey. If your survey has multiple sections or parts, then it is good to introduce each new section as suggested previously. Keep in mind to make the sections and questions flow in a sequential order that makes sense to the respondents (Brace 2004, 154).
Smart Survey Design 2. Layout for coding and identification: As the designer of the survey, pay attention to the physical layout of the survey to reduce the likelihood of errors on the respondents‟ end or on your end regarding areas of coding or editing. Here are some principles to follow to make the survey logical for all people accessing the survey, as well as easy to identify, code, and store: 1. Identification – You can add a unique number or identifier to each questionnaire.
Smart Survey Design D. Pre-Test: The pre-test or test pilot of the survey provides two functions. It first serves as the initial “live” test of the survey, and secondly it is the last step in finalizing the survey questions and form. The pre-test is possibly one of the most critical steps in administering a survey. By opting out of conducting a test pilot, you could jeopardize the accuracy of the data.
Smart Survey Design III: SURVEY RESPONSE RATES & PROS VS. CONS OF ONLINE SURVEYS A. Survey Response Rates (Taken from: http://www.utexas.edu/academic/diia/assessment/iar/teaching/gather/method/surveyResponse.php?task=research) 1. Calculating the Response Rate: The percentage of people who respond to your survey is considered the “response rate.” A high survey response rate helps to ensure that the survey results are representative of the survey population.
Smart Survey Design statistics. For a simple explanation of the difference between descriptive statistics and inferential statistics please visit the following site: http://www.statisticallysignificantconsulting.com/Statistics101.htm Inferential statistics allow you to find out if your results are “statistically significant” or not. For example, you might develop an opinion survey and measure basic demographic characteristics of the study participants such as age and gender.
Smart Survey Design 3. Maximizing Response Rate: Here are some tips for maximizing the Response Rates (Taken from: http://www.utexas.edu/academic/diia/assessment/iar/teaching/gather/method/surveyResponse.php?task=research): 1. Request participation in advance. 2. Include information regarding the following in the Introduction of the survey or in the cover letter: State the purpose of the survey. Give a reason for participation.
Smart Survey Design B.
Smart Survey Design Interesting and good graphics are important elements in paper based surveys, but an online survey can truly present well laid out and visually pleasing survey designs. Web surveys can utilize colors, images, and hyperlinks. Web surveys are also dynamic, which means they can provide statistical results on an immediate basis. Our tool provides statistical results immediately in the Analyze section as soon as a respondent submits his/her survey.
Smart Survey Design out emails back into any lists. If a respondent mistakenly opts out, then we do offer a way for him/her to opt back into the list(s) on our homepage through the Opt Out/Opt In link. Non-delivered email or messages rejected as SPAM can sometimes be a concern for survey authors. We do provide our IP addresses to give to your IT department if internal email invites are being blocked. This will help you configure your server to allow the emails to be received.
Smart Survey Design V: COLLECTING DATA AND PRIVACY BEST PRACTICES 1. Inform your respondents about how their responses will be handled. It is always good to disclose your privacy practices to your respondents. Doing this helps to increase response rates by putting potential respondents more at ease. You can disclose your privacy practices on the introduction page of your survey, or in the body of any email message containing the link inviting recipients to take your survey.
Smart Survey Design V: SECTION 508 & ACCESSIBILITY 1. Making your Surveys Accessible and 508 Compliant SurveyMonkey is a Section 508 compliant online survey application. We ensure that, by using our standard survey designs, your survey will meet all current US Federal Section 508 certification guidelines. Section 508 of the Rehabilitation Act of 1973 is a federal law that outlines requirements for making online information services accessible to users with disabilities.
Smart Survey Design in accessibility whose efforts have helped create many of the 508 standards in use today. You can get more information about Jim Thatcher by clicking here (http://www.jimthatcher.com/index.htm). With their help in design, consultation and implementation of the latest web accessibility standards, we have created user-friendly survey formats accessible with a broad range of disability software. B.
Smart Survey Design c.) Clearly outline instructions in your Question text field for questions that have specific response requirements. - Clearly state how many answer choices the respondent will need to select when creating multiple answer question types. For example, “Please select all options that apply to you” or “Please select no more than 3 options”. This helps clarify what respondents should do while they listen to the list of answer options. It is also helpful to note if the question is required.
Smart Survey Design VI: CONCLUSION From the suggestions and examples provided in this guide, you can see that creating effective surveys does not have to be an intimidating task! Everyone is capable of designing surveys to gather specific data as accurately as possible. You do not have to be a master designer to accomplish these goals, and the tips and basic guidelines provided here will help you get started in creating effective and accurate surveys.
Smart Survey Design As the account holder, you will still be responsible for administering the survey and analyzing the data. If you have additional questions in regards to these areas, please don‟t hesitate to contact us through the email support center. Now it is time for you to begin creating surveys! Feel free to contact us through our email support center if you have any questions about creating your surveys.
Smart Survey Design Appendix 1 For additional information regarding survey design and writing effective questions, please refer to the following sites: A. Survey Question Types: 1. Ezine Articles.com: Closed Ended Questions Russell, Michael. "Data Acquisition: Closed-Ended Questionnaire Response Format." EzineArticles 01 December 2006. 28 February 2007, from http://ezinearticles.com/?DataAcquisition:-Closed-Ended-Questionnaire-Response-Format&id=373764. 2.
Smart Survey Design 1. Statistical Consultant for Doctoral Students and Researchers - Plus, Statistics Tutorials Creech, Steve – President (2007). “Sample Size”. Retrieved March 1, 2007, from Statistical Consultant for Doctoral Students and Researchers: http://www.statisticallysignificantconsulting.com/Sample-Size-Help.htm 2. University of Iowa: Sample-Size Determination Lenth, Russell V. (March, 1 2001) “Some Practical Guidelines for Effective Sample-Size Determination”.
Smart Survey Design WORKS CITED Brace, I. (2004). Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research. London: Market Research in Practice Series. Creech, Steve – President (2007). “Sample Size”. Retrieved March 1, 2007, from Statistical Consultant for Doctoral Students and Researchers: http://www.statisticallysignificantconsulting.com/Sample-Size-Help.htm Crown copyright 1998. (1998, July 24). Data Protection Act 1998 chapter 29.
Smart Survey Design RampWeb Web Accessibility Consulting. 2002-2007: http://www.RampWEB.com. Sheehan, K. (2001). E-mail Survey Response Rates: A Review. Journal of Computer Mediated Communication, Vol 6 (2) . Waddington, H. (2000). Types of survey questions. In B. Hoffman (Ed.), Encyclopedia of Educational Technology. Retrieved February 28, 2007, from: http://coe.sdsu.edu/eet/Articles/surveyquest/start.htm Warwick, D.P. & C.A. Lininger. (1971). The Sample Survey: Theory and Practice. New York: McGraw Hill.
