Quasi-experimental vs Factorial Strategies in I/O Psychology

Published Date: 28 Mar 2018

Research design: Quasi-experimental vs. factorial strategies in I/O psychology

• Daniel Johnson

Quasi-Experimental

I chose a cross-sectional quasi-experimental research design that examined the effects of implementing a time management training program to improve e-mail self-efficacy, e-mail-specific time management, perceived time control over e-mail use, and estimated time spent using e-mails in order to reduce workload and increase productivity. The potential vicissitudes on perceived time control and tine spent assessment scores are theorized to be caused by e-mail self-efficacy and time management, which are referred to as the mediating variables.

As for operational definitions, E-mail self-efficacy (SE), is one’s judgment of his/her ability to perform e-mail tasks efficiently using time management strategies. Therefore, SE determines how comfortable and confident one is in using e-mail functions and is mediated by training. Therefore training was designed to increase SE, while e-mail self-efficacy was assessed using the self-efficacy five point Likert-scale developed by Compeau and Higgins. E-mail-specific time management behavior (TM), in which Time management is defined as categories of behavior (time assessment, planning, and monitoring behavior) that are “aimed at the effective use of time while undertaking goal-directed activities” (pg. 524); TM was measured by the degree to which subjects exercised various forms of time assessment, planning, and monitoring concerning e-mail use. High scores on the time management behavior (TMB) scale, which was used to measure general time management behavior with three subscales: goal-setting tendencies, time management mechanics, and organizational preferences, indicated that the participant was applying time management principles more often. Perceived time control over e-mail use (TC), measured the degree to which participants perceived their ability to limit time spent on e-mail; TC was assessed using Macan’s (1994) scale of perceived control of time to the context of e-mail use – a five-item 5-point scale. Estimated time spent in e-mail (TS) measured subjects’ assessment of time spent in e-mail and examined whether or not the perception of better time control was a reflection of actual time. Therefore, the researchers judge that TC and TS would be the results of the first-level variables, SE and TM.

Hypotheses

1. Training will improve e-mail time management behavior.
2. Training will improve e-mail self-efficacy.
3. Training will improve time control concerning e-mail use.
4. Training will improve time spent on e-mails

As mentioned above the design used was cross-sectional quasi-experimental; moreover, The nature of quasi-experiment non-equivalent control group design calls for ANCOVA analysis to include baseline scores as a covariate, which eliminates the effect of non-equivalent control group error. Subjects were recruited through the training and education unit of the human resources department, among the employees whose job functions characterized them as knowledge workers. The recruitment was presented to employees as a regular in-house training and education program. As the subjects could freely choose to attend the training sessions, it was not possible to randomly assign each subject to an experimental or control group; it was only possible to randomly designate subjects of an entire session to either an experimental or control group. Therefore the subjects in the two groups were unequal in number. Also, the nature of quasi-experiment does not assure equivalent control group (i.e., the characteristics of the control group may not be similar to the experimental group).

Specifically, there were 175 participants in the experimental and 105 participants in the control group, respectively, data were collected before and after the experiment. Baseline scores were obtained, the experimental group received the treatment, and then both groups were tested again on the same measures 1.5 months later with the test items scrambled. Since this study took place in an organizational setting and used real workers and not students, analysis of covariance (ANCOVA) was conducted to test the training effects. Scores were compared and the results indicated that the training program improved SE, TC, and TS, but not TM. Therefore, all hypotheses were supported except for Hypothesis 1.

The author addressed threats to validity concerning: History and maturation effects, Instrumentation effects and testing effects, and Hawthorne effects, respectively. History effects were addressed by the authors by identifying that their study exercised an experimental/control group and a baseline/follow-up design, thereby eliminating history effect. Additionally, the experimenters followed-up 1.5mos later thereby decreasing maturation effect. Instrumentation effects and testing effects were addressed by identifying that the baseline and follow-up question contain the same test items and were scrambled ti hide any familiarity between to two test assessments. Hawthorne effect was controlled for by clearly stating that the data will not be distributed and also by collecting evaluations regarding the instructors from participants.

Factorial Design:

The factorial design I chose compared three training methods to improve attention management skills in process control. “Process control operators must learn to apportion their attention strategically. Attention is strongly associated with mental workload. Workload is on the one hand determined by task with varying difficulty and priority”. The aim of the research goal was to support novice learners by providing them with attention management strategies in order to reduce their mental workload. Internally, the training conditions were: emphasis shift training (EST), situation awareness (SA) training with EST, and drill and practice (D&P). The first is emphasis shift training (EST), which consist of strategies, response schemas, and the voluntary control of attention. Once a schema is developed, the operation of it is assumed to require few attentional resources; consequently, multiple changes in the priority on components of a task are introduced, but the whole task is left intact. Only the attention status of the subtasks is changed. The second is situational awareness (SA) training with EST. Situational awareness is understood to be the perception and comprehension of elements in the environment and the projection of their status in the near future. SA is achieved by recognizing critical cues in the environment related to schemas – or mental models – which are developed through training and identifying recurrent situational components and causal relationships. “This approach calls for randomly stopping, or freezing, a simulated task, and asking the participant questions about the training”. The last training approach is drill and practice (D&P), which is the continuously rehearsal of a task, gradual improvement in time-sharing performance and divided attention are expected.

The operator’s task is to intervene if necessary. This individual must complete two primary tasks (system control and fault diagnosis) and two secondary tasks (prospective memory and reaction time). The levels of the independent variables were process control environments, test time dates, and fault states. The first part of the experiment consisted of questionnaires and a training module, which lasted about 4 hours, followed immediately by a 45-min testing session (test₀). The second part was a 45-min testing session two weeks after the training module (test_2w). The third part was a 45-min testing session six weeks after the training module (test_6w); therefore, a 3 × 3 × 2 mixed factorial design was used – Training (EST, EST/SA, and D&P) × Time (test₀, test_2w, and test_6w) × Fault State (familiar, non-familiar).

Hypotheses

1. The EST/SA group would be more successful than the other groups at developing and maintaining SA, since they receive SA training based on the freezing-and-debriefing technique described above, aimed directly at improving SA.
2. EST and EST/SA would be more effective than D&P at developing participants’ performance on system control.
3. Third, D&P would be more effective than the two other training methods for developing participants’ performance on diagnosing familiar fault states.
4. EST and EST/SA would be more effective for developing participants’ performance on diagnosing novel fault states than on diagnosing practiced ones.

The results for this study were interpreted using mixed ANOVAs. D&P served as the baseline measurement, while Time of Measurement was a within-participant variable, which was investigated in three separate testing sessions (test₀, test_2w, and test_6w) along with Fault type, practiced or novel, respectively. Groups differed by training approaches and fault states, which were familiar or non-familiar. A significant interaction effect, however, between fault type and training group was detected; The D&P and EST groups performed better than the EST/SA group during practiced faults, and the EST/SA group performed better during novel faults. This interaction indicates that training groups differed in whether they were either experienced or novel.

The study was conducted with students, and was therefore, not generalizable to operators working in process control. Forty students (83.3%; four female) completed all three parts of the experiment, which were all enrolled in a program leading to a Bachelor of Science degree in a technical field of study. The researchers did, however, control for cognitive ability, cognitive flexibility, conscientiousness, and pertaining motivation. Cognitive measures pertaining to reactions to training which included: Effort, Anxiety, Fatigue, Motivation, Self-efficacy and therefore were engaged to control for potential ‘‘unhappy randomization. Cognitive ability was assessed with the Wonderlic Personnel Test, cognitive flexibility, with the Cognitive Flexibility Inventory, and Big-Five Markers measured conscientiousness as a personality trait. Finally motivation was assessed with one item which indicated how motivated participants were to participate in this training – this was measured using percentile rank scores.

The authors noted that threats to validity could be addressed “methodologically, by increasing the sample size” (pg. 984). However, the authors did not address the issues of the gender discrepancies – there was only four females, likewise, as mentioned above, the majority of the participants were students and not process control operators. In addition, the authors mentioned that the retention interval between the testing sessions could not be identically long because of the students’ obligations to examines.

The researchers stated that the goal of the study was to “support novice learners by providing attention management strategies in order to decrease their mental workload,” (pg. 976), however, the introductory training module CAMS (Cabin Air Management System), the artificial task used as a in the experiment, is not known outside the scientific community and is situated (modeled after a life support system onboard a spacecraft) in the context of spaceflight, therefore, it does not correspond directly to real world physical principles.