On Reviewing and Receiving Reviews

Evaluation is not just a matter of threading a judgement through a procedure, but also of characterizing the quality of something. — Stefan Hirschauer: Editorial Judgements

Double-blind peer review is one of the pillars of the scientific apparatus. Many manuscripts are evaluated by their peers before they are being published in order to ensure quality standards and constantly redefine the scope of a field.

I mostly enjoy reviewing — no matter the quality of the paper –, since I can contribute to an idea and improve a manuscript. However, sometimes, there is this sneaky paper, where you can see that someone has been left alone by their peers. Those manuscripts have obviously not received any love or critical feedback and are rough 2nd or 3rd drafts of writing. I love discussing the content, but if I’m overwhelmed by the amount of grammar, style and structure issues I need to mention I feel like someone used the review process for proof reading and editing feedback more than actually expecting to get published. Doing this type of work for friends and colleagues is ok and I offer it willingly, but having the job dumped on me feels weird. Now, I could always reject reviewing a paper, but then I feel like I’m just deflecting the job to some other poor soul who has to go through the motions regardless. One solution there would be that publication venues offer mentors who are willing to proof read papers of more junior writers and give them structural feedback. Most academic writing is published in English, which privileges native speakers. For grammar and style issues — mostly from non-native speakers –, we should think about having an optional editing stage before or after reviewing — and communicating that to reviewers. These things cost money and time, yes, but then publishers make tons of money from work they don’t pay for anyway, so why not invest into that, so that we all can enjoy reviewing papers more and getting enthusiastic about ideas.

There is another side to reviews as well: the receiving side. I mostly enjoy receiving reviews — no matter the result –, since they always help to improve an idea as well as the manuscript and, ultimately, that’s what I want: the best version of a manuscript possible. Some reviews are unhelpful, yes, some reviews didn’t quite understand what you were writing about, yes, but in the end it tells you, that you have to be more concise in your writing. However, sometimes we receive reviews that are unkind and unappreciative towards the work you describe in your manuscript. Sometimes we receive reviews that tell us we chose not only the wrong venue (fair enough), but also that we have no place in the chosen field. This seems awfully ignorant of the fact that almost no publications come from authors who are not embedded in public or private research institutions in which they have to work with others who are also part of their field. The anonymity of authors and reviewers alike during the review process is essential to counteract the most obvious favouritism. However, often authors are already obvious to reviewers during the process or become obvious when a manuscript is published. Reviewers remain anonymous other than when they choose to identify themselves to the authors after publication and even that is frowned upon. I argue that anonymous reviewers behave somewhat like anonymous commenters on the internet: they are ultimately harsher. Reviewers are not accountable for what they write and, hence, sometimes don’t filter their criticism with the appropriate appreciation for the work in front of them. Making it standard that reviewers are disclosed after the review process is finished and a decision has been made, encourages them to stay more focused on their evaluation task and creates a more helpful and supportive review system that we all can profit more from.

Person-First or Identity-First?

When I started my PhD, I was occupied for several months with almost exclusively the question on whether I would say “autistic people” or “people with autism.” Before I say more, you should know that I do not identify as autistic myself – albeit disabled – but grew up with an autistic sibling.

I stumbled across the question of language in the way that most researchers probably do, in that I had never really thought about it, since ‘it does not concern me personally, right?’. Through that I kind of offended a friend of mine who has a kid with autism and explained to me that there is a difference in person-first or, as they called it, ‘label-first’ language. Not wanting to offend anyone, I dove into the debate. And it’s an intense one.

The main argument for person-first language is that by othering and labeling individuals through adjectives, they become second to their label and are not seen as individual. Others have elaborated on this more, but that seems to be the gist of it. If you say ‘a person with autism’ it becomes more about the person than about the autism. But then, does it?

According to Jim Sinclair, this way an autistic person is also seen as separate from their autism, which is contrary to their experience. Why can they be called clever and handsome, but not autistic? Why is it not a person with smartness or a person with beauty?

Ultimately then, it does not seem to matter how we talk about a condition grammatically, but rather, which semantics are attached to it. For those who are into that, there is some research on how the grammatical construct does not necessarily alter semantic meaning.

However, those who connect the construct with a certain meaning care very much about how we talk about autism and autistic individuals. You might have noticed by now that I am mostly using identity-first language and those who know me also know that I strongly advocate for using it. This is not only because I find Sinclair’s argument slightly more convincing, but also because a large survey in the UK indicated that autistic individuals tend to prefer it as well. Sue Fletcher-Watson recently analysed the results of the survey and shows how professionals dealing with autism according to the numbers care emphatically more about which language is used and strongly prefer person-first constructs whereas autistic people just indicate a slight preference. The conclusion then is that mixing grammatical constructs might be the least offensive way for everyone involved. I disagree.

I perfectly understand the need to work with the autism community at large, which includes autistic individuals, their families, their carers and professionals. But what is language if not a political statement. If there is a debate on how language is used and a large portion of the people we try to address is telling me ‘please call me autistic’, then who am I to judge them wrong? I need to listen to them and I need to listen to them more than to professionals who do have more power in this conversation already. Using identity-first language to me is a political act that acknowledges the preferences of those who are marginalised all too often.

This is my conclusion, but this is not the only way to do it. I don’t think there is a ‘correct’ way to talk about autistic individuals. First of all, I also change my language if I talk about an individual who prefers e.g., person first language (see above when talking about my friend’s kid). Second of all, I don’t think it helps anyone if people just try and do what is ‘politically correct’ without thinking about the circumstances that does make something correct or incorrect. That is why I might ask you why you are making a certain choice. I want to learn more about the different arguments surrounding the language and re-evaluate my own use. For me, it is most important that I make an informed choice in how I speak, especially when speaking about something that is outside of my personal experience, such as being a non-autistic researchers talking about autistic children.

Ultimately, and I think that is where everyone in the debate agrees on, consider the preference an individual has when talking about them specifically. Only that way we can start establishing a person-first attitude regardless of the grammatical constructs we use.

OutsideTheBox

For two years now I have been a PhD student with the OutsideTheBox project, in which we co-design technologies with autistic children. Our prototypes are supposed to make sense in the children’s lives and enable them to share the positive experiences they make with the technologies. My PhD work focuses on the experiences the children have with their technologies.
To that effect, I’m developing an evaluation tool that allows for critical reflection and purposefully includes the direct perspectives from autistic children. It is based on Actor-Network Theory and Critical Discourse Analysis.

Illustration

illustration drawn by Julia Makhaeva

Multicoloured Hourglass

2015-02-01 11.57.28

This is just a colour pattern, that I used as a substitute in a Norwegian Sweater, but I wanted to share that with peeps in order they want to conduct a similar feat. It has been created with a sense of symmetry and triangles. It came originally not with the hourglass idea, but upon actually knitting it, I realised it kinda felt like hourglasses with a top. Anyway, I hope you’ll have fun or are inspired by this.

2015-02-01 11.57.24

This pattern is only offered as a graphical pattern, because all stitches are knit (or purl on the way back) in three colours of your choice (marked as gray, white and black).

triangle_sandwatch

Quick Mittens

2014-12-21 13.35.51
Last year my sister got a loop made with Manos del Uruguay’s Silk Blend (Luisa).

loop
loop

This year I also made her a matching hat of from the same wool upon request as a thank you for proof reading my Master’s thesis.
Hat picture
Hat

This weekend I had a visit from her and told her that there was still a hank left, so if she wanted to give it back or whatever. But instead, she asked for mittens. And because that’s a challenge, I have to accept, I promised them within 24 hours including making up a pattern for her and all. However, I now want to give that pattern to you as well, so that you can also have an instant gratification mittens experience.
2014-12-21 13.36.13
So for these quickies you’ll need:

  • Set of 3.5mm double pointed needles (dpn)
  • 40g of DK yarn (If you want the tactile qualities of the fabric to be more visible, you might want to choose a yarn that is more evenly coloured, though)
  • Stitch markers (if desired)
  • Tapestry needle

2014-12-21 13.33.48
Abbreviations:

  • k – knit
  • p – purl
  • k tbl – knit through back loop

2014-12-21 13.34.16
Pattern Notes:

  • 1×1 rib – (k tbl, p) and repeat that all round
  • flowery weave (Variation on the Basket Weave taken from the Potter Craft 400 stitches stitch dictionary) – do 12 rows as follows over a multiple of 8 stitches (or confer for a visual reference down below)
    1. k all stitches
    2. like row 1
    3. (k3, p2, k1, p2)
    4. (k3, p1, k1, p1, k1, p1)
    5. like row 3
    6. like row 4
    7. like row 1
    8. like row 1
    9. (p1, k1, p2, k3, p1)
    10. (k1, p1, k1, p1, k3, p1
    11. like row 9
    12. like row 10

flowery_weave
How to do it (Finally!):

  • Cast on 32 stitches and distribute on needles as desired; if you want a larger or smaller version, you can cast on 8 more or less and follow this pattern just having 8 stitches more or less in the end
  • 1×1 rib for five rows
  • 2.5 repeats of the flowery weave pattern
  • in the next round (it’s a knit round) place stitch marker (if desired) after 7 stitches and then again after 2 stitches
  • in the next round (still a knit round following the pattern), make one left and right from the two stitches between stitch markers (this is an increase round)
  • Follow pattern for 2 more rows
  • repeat increase round
  • Follow pattern for 2 more rows
  • repeat increase round and do so every second round until a total of 10 increase rounds have been worked; not that for the pattern to work, you will have to incorporate one of the added stitches on each side. The rest of the thumb is knitted flat.
  • knit one round (this should coincide with a first round of the flowery weave
  • next round: knit 8, bind off 18 added stitches loosely (I use this method, but only on every second stitch, to get the results shown in the picture) and knit rest of stitches
  • 2014-12-21 13.36.43

  • from this round on you will finish this repeat of the flowery weave
  • In order to tighten the connection at the thumb you might want to knit the edge stitches together with their neighbours on the left and right from one row under (If this needs further clarification, I am happy to provide a more detailed description, so please let me know in the comments.)
  • when pattern repeat is finished, knit 2 more rows
  • 1×1 rib for 4 rows
  • Bind off all stitches as before
  • Make a second glove
  • Weave in ends
  • Block if desired
  • Be happy and proud!

2014-12-21 13.35.07

I hope you have fun doing this pattern. If there are any questions or comments, please don’t hesitate to put them in, so that I can improve this pattern.

Framing Lenses – My Defence

Monday, November 10, 2014, I have successfully defended my master’s thesis and can now call myself M.Sc.

success kid

That is great and of course I want to share this with the world. So, my full thesis can be found here and it’s a great read, but I do understand if that is too much for you. So alternatively, you can just look at the presentation or decide to read further on when there is the text of my defence. Note that there has been no adjusting of punctuation, since the text was supposed to be spoken, not read.

Welcome to my Master’s defence, the final step of the Computer Science and Media programme.
Here, we will discuss my thesis, which has been my main occupation this year. As you can see, I have been investigating questions about the Framing of Gameplay Experience in Games with Eye Movement Based Adaptation.

With this work I tried to find out, whether there is a placebo effect for games that are presented with a novel technological feature.

There are a lot of questions that could be asked now, and during the time of this presentation, I will try and address some of them.`

First, I will give a short introduction of the relevant terms and explain why it could be interesting at all. Then I will show the conceptual design of the adapted Tetris game and the conceptual design of the study as well as selected results together with their interpretation. Finally, I will present some options for further research.

But let’s start with the most obvious question: What do the words of the title even mean?

Consider framing.

While the nature of framing is studied in sociology, I use the broad definition from psychology, where framing refers to the presentation of information with different, but equivalent labels.
Concretely, what I did can be seen as attribute framing. While I did not present something positively or negatively, I did highlight different attributes (and omit others).

This lead to the first part of my research question: If we suggest technological novelty, how does that fare to actually implementing it? (We will see the answer later.)

First we need to establish, what is meant by Gameplay Experience.

There are tons of expressions and words that are thrown around within academic discourse to refer to gameplay experience or aspects thereof. Some appear often: presence, involvement, flow and immersion. I will not explain them in detail right now, you can find out more with the given references, but all in all, there is a quality to playing games and digital games, that has no canonised term in academia yet. This makes it extremely difficult to measure this quality, since it is not even really defined yet.

However, I was not set out to solve this problem, so I relied on a standardised questionnaire, the Game Experience Questionnaire, because it was the best, available for my purposes and also covers most of the previously shown aspects. All other questionnaires referred to either narrative structures or overly visual experiences, which are both not given in Tetris, the game, I was looking at.

This leads to the second part of the research question: If we suggest technological novelty, how does Gameplay Experience change compared to games actually implementing it?

To continue with the inquiry about the long title, we have to look at what eye movements might tell us.

In general, eye movements can be distinguished in two large classes. Fixations and saccades. Fixations are the relatively stable episodes in which actual visual inspection happens. One of them lasts about 250 ms. Saccades are the very fast eye movements between fixations lasting about 55ms.

Eye movement analysis can also happen on a more finegrained and derived level, but they are all based on fixations and saccades. It is important to know that previous research indicates that experts have overall less fixations and larger saccadic amplitudes, this means, their saccades cover larger areas before a new fixation occurs. Relevant for the adaption processes I will show you in a few slides, are also focus areas, which accumulate fixations in an area of interest. Experts tend to have more fixations in these areas.

For Tetris this is the contour and the current tetromino.

There are also transitions, which describe how often the point of interest has changed between areas on the screen.

Here, there is an indication, that a user is more uncertain if there are frequent changes of focus. Look at the areas here and the fixations how they start here and then change in another area and stay there a little.

And there are scanpaths, which are an illustration of the direction of focus change over time. Especially for Tetris, Underwood illustrated in 2005, that experts show more horizontal than vertical eye movements in Tetris. In order to record these eye movements, eye tracking is usually done with a high frequency camera pointed at the eyes which is either located under the area of inspection or mounted on the head of the person who’s eyes are being tracked. I have used a head mounted version, the EyeLink II by SR Research, because it was the one with the highest temporal resolution available to me at the time.

With this data about eye movements, we can further elaborate on the research question. If we suggest technological novelty, how do Gameplay Experience and Expert Behaviour as measured by eye movements change compared to games actually implementing it?

In fact, I had a total of18 hypotheses which were again divided in different parts and all driven by different parts of the research question, but the main research question comes down to this,

where technological novelty is an aspect of games with eye movement based adaptation.

So let me rephrase: What has more effect on Gameplay Experience and expert behaviour as measured by eye movements: The implementation of eye movement based adaptation or simply the suggestion of this technology?

I want to explain now, why this is at all an interesting question for me.

So far I’ve only shown you a glimpse of relevant previous research and due to time constraints I am also not able to delve any deeper. There has also been our previous study on Hex, where we asked players whether they would have acted differently if informed beforehand that their eye movements are used. Most of them agreed. In the informed control group, however, more people stated that they didn’t act or behave differently. The data we had collected in this case was non-conclusive, so the question whether there is a palpable effect became more and more interesting to me.

Additionally, I am trying to do science with a critical approach I learned via the other field I studied; media culture. This means, I try to ask novel questions driven by a critical stance towards technological development. So, looking at this cool new thing we developed – in our case eye movement based adaptation – I’m trying to find out whether it makes sense on its own and how much of an effect can be attributed to the simple novelty of the technology. It appears that this question is rarely asked in sciences with a mainly positivistic view on its outcomes.

Furthermore, with the availability of comparatively cheap eye trackers with good enough frame rates and an option to grab data in real time, the development of games using those eye movements has a new momentum. Just last month the CHI Play conference had its own workshop on Applications for Gaze in Games (called EyePlay). Most of the projects conceptualised there use eye movements as an input device. While I have the feeling that this might be a bit frustrating, when imprecise measures (how they still occur even in eye trackers with well adjusted head tracking), when these imprecise measures lead to unintended inputs. However, taking more derived measures to adapt to eye movements in a stepwise or even historic fashion seems to be more promising to me as single eye movements only contribute partly to an accumulated effect.

There are several ways to accomplish this, let me show you my version of eye movement based adaptation.

I have used Tetris in my research, because it is visual and also spatial. Next to the obvious modifier of difficulty — the speed in which tetrominoes, the Tetris pieces, appear, the random choosing algorithm that decides which piece comes next can be altered. I did a small pre-study with 16 participants in order to find out, which algorithms are more difficult than others. It turns out, that a lot relies on the frequency of s and z pieces occurring – the zigzaggy ones noone appears to like anyway. With this knowledge I had a second way to adapt Tetris.

In order to have a control version, that is adaptive but not using eye movements, the fully adaptive process uses components with and without eye movements jointly. This makes it possible to later check whether there is an effect of eye movement based adaptation that cannot be attributed to adaptation in general. To adapt to algorithms, I have assigned numbers to them. The non-eye movement based adaptation mode is bumpiness. It describes the variance in height differences along the columns of a current game. The transition value tells us about the confidence with which a player is playing. This means, the more transitions over time, the less secure a player is. High bumpiness as well as a high transition value lead to an easier algorithm. However, the rationale here is that even if a player plays comparatively well, if they feel insecure about their performance, the difficulty should be adjusted differently.

For speed adaptation, I chose pile height. Remember the original Tetris game, pile height is the measure by which speed is adjusted. Every time, the pile height is reduced, the speed is increased. In my version, as soon as a player reached a pile height of five for the first time, I decreased the speed a little bit. The increase still happened faster than the decrease in order to not completely counteract a player’s expectations in how the game works. Then the eye movement based speed adaptation checked how often players looked in areas of special interest. This means the more focused players were, the faster the speed.

In order to see, what the effect of eye movement adaptation really is compared to its suggestion, I needed four groups. Players who were framed as in told that their eye movements were used for adaptation had to play games with eye movement based adaptation and games without. Players who were not framed needed to play both games as well. Since in the pilot study players were not able to distinguish the different types of adaptation, I was able to divide the real test persons in groups according to framing and letting both groups play both types of games. This means, of my 43 test participants 22 were informed about the use of eye movements whereas 21 were not informed. Each participant played 2 games with eye movement based adaptation and 2 games without. During all games their eyes were tracked, which was important for the results, but also to not give an external indication about which type of adaptation was used.

So, what were my results?

This graph shows means and the error bars depict one standard deviation from that mean. This is the case for all of the following figures as well.

First of all, I could show that the games are significantly different when eye movements are used in the adaptation process. The average value of the algorithm is higher in games with Eye Movement Based Adaptation, leading to a harder game. Harder algorithms produce less fitting pieces; hence, it is harder to make lines.

The speed was also significantly different between the two modes of adaptation. Conventionally adapted games ended on a higher speed than games that also used eye movement based adaptation. This means, that the pressure during the game came more from the speed at which the pieces appeared and fell down instead of ill-fitting pieces in conventionally adapted games, whereas in eye movement based games, the difficulty came from the shapes more than from the speed at which the pieces fell down. All in all it can be established that the adaptation via transition values indicating uncertainty of players overall added to the difficulty since it was solely targeted at the piece choosing algorithms whereas the analysis of expert fixations lead to an overall easier game. Seeing as the performance did not differ significantly across the modes of adaptation, it was interesting to me to check whether there has been a different experiential quality to the games.

In conventionally adapted games, framed players reported a significant difference in experienced challenge. Note, just the framing was different. This difference in experience is also not reproduced for eye movement based games, which indicates that the expectations of framed players shaped their perceived challenge. This does also indicate that games with eye movement based adaptation are expected to be more difficult.

It is furthermore interesting to see that framed players also perceived their competence differently according to which game they were facing. In games with eye movement based adaptation they reported significantly more competence than in conventionally adapted games. This means, that when their expectations were actually met, as in they expected a game to use their eye movements and it actually did, they felt like their strategies were having a bigger effect, which cannot be established by the actual performance measures.

Finally, there is also a significant difference in expert related eye movements at least in conventionally adapted games. This measure has not been analysed for games with eye movement based adaptation, since there for those that is an input variable. So, in conventionally adapted games, framed players show significantly less fixations outside of the scope of the interesting regions than non-framed players. This indicates that framed players were more concentrated on relevant points of the game.

Thinking back at the research question. Originally I was set out to determine what has more of an influence on players: The novel technique of eye movement based adaptation or its simple suggestion.

Taken these findings together, it turns out that eye movement based adaptation as well as framing have different effects on players. One without the other is less influential than both together; but personally, I would consider it unethical to only have one anyway.

What can we now do with these findings?

Well, first there are things that could be done better. Not only is it probably more desirable to use a remote eye tracker instead of mounting a heavy piece of hardware on a players head, but, it it also important to incorporate qualitative methods in a similar study. I have no way of telling you about the story where I had brother and sister in my lab playing at different skill levels but being equally impressed by each other’s performance and having similar levels of fun; I cannot tell you this story in a structured fashion because I had no methods recording this reliably and my quantitative measures don’t suffice to describe experiences adequately. Also the expectations framed players had are not recorded although they have often been verbalised before the game or even during.

Finally, there is also further research that can be conducted from here. Other games could be interesting, especially more complex and dynamic ones. Hex and Tetris rely heavily on turns and episodes, maybe a fluid game produces different eye movements. Also the development of eye movements over several episodes could be interesting. This could also lead to a prediction of likely eye movements in certain situations and with an algorithm anticipating those, eye movements could be lead whenever that is suitable (e.g. by making areas pulse in order to draw attention to them). It might also be fruitful to consider different patterns of eye movements. For example people with autism look at different areas and different details in an image. If the presented visual structure of a game is more complex than Tetris, this might play a role for gameplay experience.

However, I think I have been able to contribute to our knowledge in two core ways: I have done an investigation into framing effects of the suggestion of technological advance and with this I have done critically informed quantitative research. Furthermore and in order to do so, I have conducted an extensive user study of which I have shown you an overview of the results right now. I hope, this is a contribution that adds to science beyond the fact, that I personally might achieve an academic degree from it.

That was it from me for now. Thank you for your attention. Are there any further questions?