Projects / Programmes
Development and evaluation automation of survey questionnaire
| Code |
Science |
Field |
Subfield |
| 5.03.00 |
Social sciences |
Sociology |
|
| Code |
Science |
Field |
| S274 |
Social sciences |
Research methodology in science |
| Code |
Science |
Field |
| 5.04 |
Social Sciences |
Sociology |
Development and evaluation automation of survey questionnaire, Social science research, Internet research, Web surveys, machine learning, language tecnologies
Researchers (16)
Organisations (2)
Abstract
Surveys are an essential research tool in the social sciences. Their quality crucially depends on questionnaire development, which is a very complex process requiring various types of knowledge, from theoretical conceptualisations and cognitive aspects through to survey methodology and statistics.
However, despite rapid developments in information and communications technologies (ICT) in the last few decades the questionnaire development process has hardly changed. This is in strong contrast with survey data collection itself which has been profoundly transformed by the integration of ICT and become largely computer-assisted.
The proposed project provides a breakthrough in integrating ICT into the questionnaire development process. It explores the potential to combine methodological knowledge with machine learning and language technologies so as to improve questionnaire development and move the entire process from text processors and off-line interactions to one integrated web platform. Research activities will take two directions:
1. The decomposition and formalisation of the question development process, and its computerisation, from the initial defining of constructs to the final wording of questions. The potential of machine learning and language technologies will be studied for providing interactive assistance within each of these steps (e.g. outlining and structuring concepts, suggesting sentence structure, offering the wording, selecting the scales, etc.).
2. The investigation will include the identification of question wording problems, which can reduce various aspects of data quality. The project will systematically explore the potential to develop and integrate methodological knowledge with machine learning and language technologies in order to interactively handle a large spectrum of wording issues, from uncommon words, ambiguous wording, low readability to double negatives, etc.
The applicable research findings described above will be developed as prototype features within existing open-source web survey software. It is there that the guidance, recommendations, suggestions, warnings etc. will be embedded into a real-time process of questionnaire development. The implementation will be done for Slovenian and English, with possible extensions to other languages.
The project will result as (i) an improvement in the speed and efficiency of the process; and (ii) an increase in the corresponding survey data quality. Both aspects will be empirically verified in experimental evaluations.
This new and innovative approach to questionnaire development will foster global research at the intersection of survey methodology, artificial intelligence and language technologies. The results will demonstrate the potential of modern ICT in this field which will, in turn, also stimulate the industry, i.e. web survey software developers, to improve their tools.
The coordinating team at the Faculty of Social Sciences, University of Ljubljana, includes renowned pioneers in web survey methodology research since 1996 and authors of highly cited scientific work. In 1998 they established the WebSM portal, a central global resource for web survey methodology research. Since 2008 they have also been developing their own open-source survey software tool.
The development of algorithms will be conducted jointly with the Artificial Intelligence Laboratory at the Jožef Stefan Institute, which is one of the leading global experts in machine learning and language technologies.
Preparatory activities will be run in close cooperation with Prof. Willem Saris (University Pompeu Fabra) who has established a conceptual framework for the construction of survey questions (Saris et al. 2007a, 2014), which will be used here. The question wording issues will be addressed in close cooperation with Prof. Jon Krosnick (Stanford University), a prominent researcher in the field.
Significance for science
Automatization of the processes of the development of the survey questionnaire is of great importance in the context of empirical social research. On this basis, solutions can be created that help to improve the quality of survey questionnaires, as well to save time and time and financial resources. In this context, the project contributed to new basic insights in understanding of the process of development of the survey questionnaire. This involves a clear conceptual separation of the selection of the type of a question and its layout, which have with the computer assisted surveys become fully intertwined and inseparable. Such separation can be a good starting point for a more effective sequence of activities in the process of developing each single question. Similarly, in the field of basic research, there is an important contribution to a greater integration of language technologies into the process of developing the survey questionnaire. The integration is in real-time, which includes interactive guidelines for the design of a particular question. The project thus represents one of the first attempts to include the achievements in the field of modern language technologies into the design of the survey questions. The project also brought direct benefits to the field of survey research and also to broader social research. The lack of integration of the potentials of information and communication technology into the process of development of the survey questionnaire means, that the collected data in empirical social research does not reach the quality that it could be reached. At the same time, this means lower efficiency, which means the costs are too high and too much time is spent. Two specific aspects apply here: • The conceptual separation of the selection of the type of survey question opens new possibilities for adaptation of interfaces to the tools for survey questionnaire development. This is primarily an improvement in terms of a deeper reflection of researchers on methodological decisions and not so much about improvements in the sense of user experience (UX). • In the view of the integration of language technologies, the project has, with developing the prototype, shown that automated approaches using language corpora can to a certain extent replace existing approaches for evaluation of the survey questionnaire which are usually very expensive and time consuming (e.g. expert interview, cognitive interview, empirical implementation).
Significance for the country
In terms of importance for the developments of science in Slovenia we would highlight: - The project results received an international attention which is important for the enforcement of Slovenian scientific achievements in the world. Numerous presentations at international conferences were carried out, including an invited lecture at a major international conference. Besides scientific articles, chapters in books, we published a monograph Web Survey Methodology at publisher Sage. These achievements strengthened the position of Slovenian research group, CSI (and therefore the position of Slovenia) in the global scientific community. - Within this project CSI members intensified the collaboration with leading research groups in the world. New opportunities for networking and cooperation with most distinguished academic institutions in this field have been created. - With the new classification of survey question types (http://www.websm.org/c/1890/Onecol) the project significantly enhanced the WebSM website (www.websm.org), which has been maintained by the CSI research group for 20 years. This has further consolidated the website as a central information resource in this field of expertise. - The project is also integrated into the pedagogical process of Social informatics at all three stages, since it is directly involved in numerous courses (e.g. Survey Methodology). It is extremely useful for students to get to know the latest researchers, but above all the students at all stages were also involved in the project activities. The project involves a series of diplomas, master’s degrees and two doctoral theses. - The project is complementary with other ARRS projects that were carried out at CSI during the project implementation. It is also complemented with the activities of research program “Internet research”. Similarly applies for the complementary with international projects, including European Social Survey and the COST project WEBDATANET. - This project has an important role in organizing the leading professional event in this field in Slovenia: Web Survey Day (https://www.1ka.si/d/en/web-survey-day). This event contributes to the popularization of the profession, as it attended by over 200 attendees from public, private, research and non-profit organizations. An important component of the event is also the presentation of new research project achievements. - The project also contributed to the extension of the functionalities of open source tool 1KA (www.1ka.si) for implementation of web surveys. 1KA has over 44,000 users and supports annually about 2 million online questionnaires. Based on the results of this project Slovenian users have more functionality.
Most important scientific results
Annual report
2015,
final report
Most important socioeconomically and culturally relevant results
Annual report
2014,
2015,
final report
