Associate Professor Marc Adam

Recent research has employed deep learning to detect intake gestures from inertial sensor and video camera data. However, the fusion of these modalities has not been attempted. The present research explores the potential of fusing the outputs of two individual deep learning inertial and video intake gesture detection models (i.e., score-level and decision-level fusion) using the test sets from two publicly available multimodal datasets: (1) OREBA-DIS recorded from 100 participants while consuming food served in discrete portions and (2) OREBA-SHA recorded from 102 participants while consuming a communal dish. We first assess the potential of fusion by contrasting the performance of the individual models in intake gesture detection. The assessment shows that fusing the outputs of individual models is more promising on the OREBA-DIS dataset. Subsequently, we conduct experiments using different score-level and decision-level fusion approaches. Our results from fusion show that the score-level fusion approach of max score model performs best of all considered fusion approaches. On the OREBA-DIS dataset, the max score fusion approach (F1 = 0.871) outperforms both individual video (F1 = 0.855) and inertial (F1 = 0.806) models. However, on the OREBA-SHA dataset, the max score fusion approach (F1 = 0.873) fails to outperform the individual inertial model (F1 = 0.895). Pairwise comparisons using bootstrapped samples confirm the statistical significance of these differences in model performance (p<.001).

Accurate detection of individual intake gestures is a key step towards automatic dietary monitoring. Both inertial sensor data of wrist movements and video data depicting the upper body have been used for this purpose. The most advanced approaches to date use a two-stage approach, in which (i) frame-level intake probabilities are learned from the sensor data using a deep neural network, and then (ii) sparse intake events are detected by finding the maxima of the frame-level probabilities. In this study, we propose a single-stage approach which directly decodes the probabilities learned from sensor data into sparse intake detections. This is achieved by weakly supervised training using Connectionist Temporal Classification (CTC) loss, and decoding using a novel extended prefix beam search decoding algorithm. Benefits of this approach include (i) end-to-end training for detections, (ii) simplified timing requirements for intake gesture labels, and (iii) improved detection performance compared to existing approaches. Across two separate datasets, we achieve relative $F_1$ score improvements between 1.9% and 6.2% over the two-stage approach for intake detection and eating/drinking detection tasks, for both video and inertial sensors.

Experiments in the field of behavioral economics often require repeated matching of participants to groups over multiple periods. Perfect stranger matching requires that no two participants interact more than once during the experiment. Computing a sequence of perfect stranger matches is an NP-hard problem that has received little attention in experimental economics literature beyond brute-force approaches. This work provides a problem definition and an algorithm for perfect stranger matching that outperforms existing approaches in the field of experimental economics in terms of problem size and number of found matches.

Internet auction sites frequently employ images as design elements on their websites in order to either induce a sense of community or competition among the bidders. In this paper, we investigate the impact of such affective images on bidding behavior in a controlled laboratory experiment during which participants' emotional processes are assessed through psychophysiological measurements. Immediately before placing a bid in a first-price sealed-bid auction, bidders are presented a) pictures of competitive sports scenes, b) pictures of families or children, or c) a blank screen. Participants place significantly lower bids when they were exposed to pictures that induce competition emotions as opposed to pictures that induce community emotions. This relationship is moderated by the bidders' emotion regulation strategy. In particular, we find that the more participants try to suppress their emotional responses to the presented images, the more they are affected in their bidding behavior. Our results entail valuable insights about the coherence of emotional stimuli on Internet auction marketplaces and customers' decisions. They also question recent marketing strategies by the market leader.

Auction sites on the Internet frequently put bidders under time pressure or highlight the social competition that is inherent to auctions. Both aspects are believed to elicit an exciting shopping experience, which may culminate in auction fever. In two laboratory experiments, we investigate the process of auction fever in retail auctions and demonstrate when and how auction fever affects bidding behavior. In contrast to previous studies, we employ physiological measurements as an objective and continuous assessment of bidders' arousal in addition to a subjective assessment of bidders' emotions through psychometric scales. Moreover, we explicitly study the interaction of time pressure and social competition on arousal and bids. We find that bidders' arousal is increased in high time pressure auctions and that this leads to higher bids in ascending auctions-but only when bidders compete with human opponents. Thus, social competition is the actual driver underlying the auction fever phenomenon. Furthermore, we show that the "joy of winning" is significantly stronger than the "frustration of losing" in ascending auctions. Finally, we discuss the implications of our findings for the design of retail auctions.

© 2013 IEEE. Emotion elicitation and classification have been performed on standardized stimuli sets, such as international affective picture systems and international affective digital sound. However, the literature which elicits and classifies emotions in a financial decision making context is scarce. In this paper, we present an evaluation to detect emotions of private investors through a controlled trading experiment. Subjects reported their level of "rejoice" and "regret" based on trading outcomes, and physiological measurements of skin conductance response and heart rate were obtained. To detect emotions, three labeling methods, namely binary, tri-, and tetrastate "blended" models were compared by means of C4.5, CART, and random forest algorithms, across different window lengths for heart rate. Taking moving window lengths of 2.5 s prior to and 0.3 s postevent (parasympathetic phase) led to the highest accuracies. Comparing labeling methods, accuracies were 67% for binary rejoice, 44% for a tristate, and 45% for a tetrastate blended emotion models. The CART yielded the highest accuracies.

Mobile health, or mHealth in short, concerns using mobile devices to provide health information to users. Research has shown that the use of avatars within mHealth applications can improve health outcomes. While several studies have investigated the design of avatars for mHealth applications, these studies focused solely on western cultures and most of them have not considered whether design principles are different between Western and Arabic users. This work will investigate the design of Arabic avatars by conducting a qualitative study, which includes semi-structured interviews with multiple stakeholders. The study will adopt a co-design methodology to derive design principles for persuasive Arabic avatars in the context of mHealth. In particular, it is our aim to increase users' perceived social presence and intention to use persuasive mHealth avatars by taking Arabic culture into consideration. The work may lead to a higher level of adoption of avatar-based mHealth applications in Arabic countries.

Management decisions are taken by human beings, not by robots. Consequently, management decisions, and of course also the respective managers, are affected by emotions. Thus, they rely on accurate emotional processing. Research on decision making has shown that individuals with high emotion regulation capabilities perform better in taking effective decisions. Managers perpetually have to take rapid decisions in fast-paced environments, between the poles of diverse interests and motives of colleagues, customers, partners, and rivals. Sophisticated management is the key to any business. Therefore, we argue that IS research should build on the advances in cognitive neuroscience and harness the potential of NeuroIS tools in the field of management support. In this paper, we propose a conceptual framework and taxonomy for how NeuroIS tools may support managers in taking effective decisions by firstly improving their emotion regulation capabilities and, secondly, providing them with real-time feedback and decision support based on physiological measurements. Based on the framework, we outline a specific application for how emotions can affect decision making in the dynamic process of negotiations and for how NeuroIS research can contribute to a better understanding of the underlying visceral processes.

Research on financial decision-making shows that traders and investors with high emotion regulation capabilities perform better in trading. But how can the others learn to regulate their emotions? 'Learning by doing' sounds like a straightforward approach. But how can one perform 'learning by doing' when there is no feedback? This problem particularly applies to learning emotion regulation, because learners can get practically no feedback on their level of emotion regulation. Our research aims at providing a learning environment that can help decision-makers to improve their emotion regulation. The approach is based on a serious game with real-time biofeedback. The game is settled in a financial context and the decision scenario is directly linked to the individual biofeedback of the learner's heart rate data. More specifically, depending on the learner's ability to regulate emotions, the decision scenario of the game continuously adjusts and thereby becomes more (or less) difficult. The learner wears an electrocardiogram sensor that transfers the data via Bluetooth to the game. The game itself is evaluated at several levels.

Information systems have revolutionized the nature of markets. Traditionally, markets inherently comprised the strategic interaction of human traders only. Nowadays, however, automated trading agents are responsible for at least 60% of the US trading volume on financial stock markets. In this respect, financial markets of the 21st century are different to markets of previous centuries. Fuelled by discussions on their possible risks, there is a need for research on the effects of automated trading agents on market efficiency and on human traders. In order to systematically investigate these issues, we introduce a market framework for human-computer interaction. This framework is then applied in a case study on a financial market scenario. In particular, we plan to conduct a NeuroIS experiment in which we analyze overall market efficiency as well as the trading behavior and emotional responses of human traders when they interact with computerized trading agents.

Although the importance of emotions in economic decision-making is being widely accepted, it is still unclear how and to what extent emotional processing affects economic behavior. In this paper, we propose a new methodological framework for behavioral IS studies that we call physioeconomics, by which it is possible to measure psychophysiological correlates of human emotions during controlled laboratory experiments. In contrast to brain imaging technologies as in NeuroIS, the analysis of autonomic physiological parameters can be achieved with a comparatively low amount of measurement overhead and, thus, allows collecting empirical data for much larger sample sizes. Therefore, this approach is particularly qualified for the study of emotions in electronic markets. In order to show that physioeconomics is able to provide insightful results we apply this method in an exemplary auction experiment in which we compare bidders' arousal throughout the bidding process and in response to winning or losing.