Research of the factorial knowledge assessment paradigm efficiency illustrated by an example of testing students in the automatic control theory

An algorithm for constructing test materials and the test results evaluation techniques that allow combining high estimate rate and test objectivity are developed. The test system in this case can be considered as an assessment tool for the knowledge level under investigating the appropriate modules. The decomposition of knowledge into lots of sections is conducted for the knowledge systematization. At that, each section is associated to a set of sub-sections which, in turn, consists of a subset of questions, the answers to which reveal the level of knowledge of each section. The test assignment is formulated in such a way that upon answer, a set of responses revealing the knowledge level of a certain subsection is to be given. Since the test development process is considered as a construction of multivariable dependence, it is proposed to approach the problem from the perspective of planning multifactor experiments. In this case, a set of answers is presented as a set of factors taking on certain values. With their varying on two levels, the problem is reduced to the construction of the experiment that implements all possible combinations of factors, i.e. to a complete factorial experiment. The technique of applying the technology of two-level multifactor experiment planning that allows formalizing the evaluation of partially incomplete knowledge, and implementing the formation of the multifactor tests of different kinds of complexity is offered. To illustrate this technique, the examples of the construction of tests for the discipline “Automatic control theory” assessing the degree of the knowledge level formation are considered. The statistical analysis of the test results has shown the knowledge level of the testees, as well as the credibility of this knowledge.

тестирование, фактор знания, тестовый вопрос, оценка ответа, вариативность ответа, неравномерность знаний, testing, knowledge factor, test question, answer mark, answer variability, knowledge irregularity

1. Yefremova, N.F. Problemy formirovaniya fondov otsenochnykh sredstv VUZov. [Generation problems of university assessment resources funds.] Higher Education Today, 2011, no. 3, pp. 17–21 (in Russian).

2. Yefremova, N.F., Meskhi, B.C. Sistemnost' i preemstvennost' v formirovanii fonda otsenochnykh sredstv tekhnicheskogo VUZa. [Systemacity and consistency in the assessment tools fund formation at the technical university.] Sovet rektorov, 2011, no. 5, pp. 33–40 (in Russian).

3. Veshneva, I.V. Razrabotka bazy modeley intellektual'noy sistemy monitoringa protsessa formirovaniya professional'nykh i obshchekul'turnykh kompetentsiy studentov vuzov. [The development of the model of intellectual system of the formation of professional and transcultural competences of universities students monitoring process.] Vestnik Saratov State Technical University, 2012, vol.1, no. 2(64), pp. 348–355(in Russian).

4. Vasin, А.N. Informatsionnaya obrazovatel'naya sreda — put' k povysheniyu effektivnosti obrazovatel'nogo protsessa. [Information educational environment – the way to educational process efficiency increase.] Vestnik Saratov State Technical University, 2010, no. 1(44), pp. 121–126 (in Russian).

5. Yefremova, N.F. Podkhody k otsenivaniyu kompetentsiy studentov pervogo kursa, pristupayushchikh k osvoeniyu osnovnykh obrazovatel'nykh program. [Approaches to the competencies assessment of the first-year students starting main educational programs.] Vestnik of DSTU, 2010, no. 5, pp. 774–782 (in Russian).

6. Yefremova, N.F. Osobennosti postroeniya regional'noy sistemy otsenki kachestva obrazovaniya. [Peculiarities of formation of a regional system of educational quality control.] Vestnik of DSTU, 2009, no. 1, pp. 762–772 (in Russian).

7. Bolshakov, А.А., Markelov, A.Yu. Kontrol' znaniy v intellektual'noy obuchayushchey sisteme po kursu «Nadezhnost' sistem avtomatizatsii». [Control of knowledge in intellectual training system at the rate ―reliability of systems of automation‖.] Vestnik Saratov State Technical University, 2010, no. 4(50), pp. 153– 156 (in Russian).

8. Bolshakov, А.А., Shatohin, V.V. Otsenka rezul'tativnosti funktsionirovaniya avtomatizirovannoy obuchayushchey ekspertnoy sistemy. [Estimation of productivity of functioning of the automated training expert system.] Vestnik Saratov State Technical University, 2010 no. 4(50), pp. 149–153 (in Russian).

9. Bolshakova, M.А. Sozdanie avtomatizirovannoy obuchayushchey sistemy dlya izucheniya inostrannogo yazyka v neyazykovykh vuzakh. [Creation of the automated learning system for learning of foreign language in not language high schools.] Vestnik Saratov State Technical University, 2010, no. 4(50), pp. 161–164 (in Russian).

10. Bolshakov, А.А., Dolinina, O.N., Shatokhin, V.V. Upravlenie obrazovatel'nym protsessom na osnove avtomatizirovannykh kombinirovannykh obuchayushchikh sistem. [Computerized teaching systems in management of education.] Vestnik Saratov State Technical University, 2008, no. 3(35), pp. 54–62 (in Russian).

11. Chertkova, Е.А. Avtomatizatsiya analiza i proektirovaniya komp'yuternykh obuchayushchikh system. [Analysis automation and computer teaching systems design.] Vestnik Saratov State Technical University, 2006 no. 1(11), pp. 97–103 (in Russian).

12. Kuznetsov, А.V. Оценивание параметров тестов [Test parameters estimation.] Vestnik Saratov State Technical University, 2006, no. 2(13), pp. 9–13 (in Russian).

13. Neydorf, R.А., Obukhova, E.N. Metodologiya organizatsii testirovaniya na osnove algoritmov planirovaniya i obrabotki dvukhurovnevykh mnogofaktornykh eksperimentov. [Testing design methodology based on two-level multifactor experiment planning and data processing.] Vestnik of DSTU, 2014, vol. 14, no. 2 (77), pp. 110–120 (in Russian).

14. Neydorf, R.А., Obukhova, E.N. Metodologiya postroeniya testov na osnove algoritma planirovaniya dvukhurovnevykh mnogofaktornykh eksperimentov. [Methodology of tests construction based on scheduling algorithm for two-level multifactorial experiments.] Informatsionnye tekhnologii v upravlenii (ITU-2012): materialy 5-y Ros. mul'tikonf. po problemam upravleniya. [Information technologies in management (ITU-2012): Proc. 5th RF Multi-conf. on management problems.] St. Petersburg, 2012, pp. 861–870 (in Russian).

15. Neydorf, R.А., Obukhova, E.N. Algoritm rascheta i otsenki rezul'tatov faktorno-testovoy otsenki. [Algorithm for calculating and evaluating the factor-test assessment.] Nauchnoe obozrenie, 2015, no. 2, pp. 41–56 (in Russian).

16. Neydorf, R.А., Obukhova, E.N. Metodika sostavleniya testovykh zadaniy na osnove algoritma planirovaniya dvukhurovnevykh mnogofaktornykh eksperimentov i analiz ikh ispol'zovaniya v obrazovatel'nom protsesse. [Test task design methodology based on two-level multifactor experiment planning algorithm and analysis of its application in the educational process.] Metodiki i tekhnologii obespecheniya i otsenki kachestva obrazovaniya sb. materialov mezhdunar. nauch. konf. [Techniques and technologies to ensure and assess the quality of education: Proc. Int. Sci. Conf.] Kiev, 2013, pp. 23–28 (in Russian).

17. Obukhova, E.N. Osobennosti formirovaniya otsenok v strokakh otvetov v dvukhfaktornom testovom zadanii. [Specific features of assessment generation in the response lines in two-factor test task.] Aspirant, 2014, no. 4, pp. 65–67 (in Russian).

18. Neydorf, R.А., Obukhova, E.N. Analiz primeneniya trekhfaktornykh drobnykh testov v uchebnom protsesse. [Analysis of the application of fractional three-factor test in the educational process.] Matematicheskie metody v tekhnike i tekhnologiyakh: sb. tr. XXVI mezhdunar. nauch. konf. [Mathematical methods in engineering and technologies: Proc. XXVI Int. Sci. Conf.] N. Novgorod, 2013, pp. 320–323 (in Russian).

19. Obukhova, E.N. Postroenie i issledovanie testovykh zadaniy pri razbienii ikh na drobnye repliki. [Construction and investigation of test tasks when partitioning them into fractional replicates.] Innovatsiya, ekologiya i resursosberegayushchie tekhnologii : tr. X mezhdunar. nauch.-tekhn. foruma. [Innovation, ecology and resource-saving technologies: Proc. X Int. Sci.-Eng. Forum.] Rostov-on-Don, 2012, pp. 363–367 (in Russian).

20. Afanasyev, V.V. Teoriya veroyatnostey v voprosakh i zadachakh. [Probability theory in questions and problems.] Yaroslavl' : izd-vo Yarosl. gos. ped. un-ta im. K. D. Ushinskogo, 2004, 249 p. (in Russian).