Chapter 2 Preparing for a stellar project

You may be joining the lab hoping to publish some of your work. We definitely aim for the work we carry out to be good enough to promote to colleagues around the world. We always want our work to be convincing and to make an impact. This section lays out our guiding philosophies for making that happen.

2.1 Choosing a research question

Researchers just beginning often struggle with choosing a good research question. That’s no surprise: this is really difficult. There are a lot of factors to consider, both in terms of logistics and potential impact. Under pressure to find a question, the junior researcher might settle for a trivial extension that does not have much theoretical depth (e.g., replicating an interesting finding in a different population). Extensions like that are not necessarily uninteresting, but they usually lack theoretical drive (e.g., is there a good reason to think that the finding would differ in that population?). The strongest research questions also lead to one of a small set of possible conclusions, not just did it or did it not work. Ideally, you want to be clear about what the competing predictions are, and develop a design where you could learn something even from finding a null effect.

It is also difficult to estimate how much time a project will take. Even with many years’ experience, I cannot do this reliably. Enthusiastically, young researchers starting their first project often come up with huge, impractical ideas that would be impossible to complete under their time and resource constraints.

You should definitely think about what research questions interest you, and you should even think grandly about what you would like to accomplish. But after you have done that, let your PI help. Your PI will have the background knowledge you currently lack to identify ways to refine your ideas and give them more theoretical depth. She will be able to suggest directions you can look into. She will know whether what you are considering is feasible or not.

The ability to hone in on a focused and interesting question will develop as you accumulate knowledge. I definitely couldn’t do it as an UG, and I knew at the time I couldn’t. To get a start on my honor’s research, I asked one of my mentors for help. After listening to Prof. Janet Kistner give an informal presentation at a lab meeting, I approached her about doing an UG dissertation (it was optional at FSU, and most US universities actually). My starting pitch was simply that I found her group’s project about the development of motivation interesting, and that I would like to learn more about it, and did she think there was any possibility to carry it further with an UG dissertation project? I wasn’t confident that I could independently identify a good way to contribute, but I could explain why this research appealed to me (it was the clever operationalization and manipulation her group used) and indicate that I wanted to learn more about it. So, you do not need to have a brilliant idea ready - you just need to think about your interests and goals enough to express them to your PI. Then together you will weigh options and refine them.

Shifting to the PGR level, you have often identified your topic before you even arrive on campus. Usually, PGRs have agreed about the broad topic of their project with a potential supervisor before even applying to the program. When you are seeking supervision as a PGR, you should be seeking someone who can evaluate the research you want to do. Never spam PIs from whole departments with your request for supervision. Do not bother cognitive psychologists with your proposal about dentistry, or agricultural business administration, or desire to master the Western blot technique (I’ve seriously received earnest, seemingly personal emails from prospective students seeking my supervision about all of these topics). Put in the effort to identify potential supervisors who actively do work in the field you are proposing to work in. You are unlikely to even get a response from the others. Even if you do, it will just be confusing, and is unlikely to lead to good supervision.

Assuming you have chosen this lab for good reasons, the first step on arrival at your PGR lab is to discuss your PGR proposal with your supervisor. The proposals we request for admitting students are short and lack much detail. They might contain good ideas, but they are not really plans of action. First thing: sit down with your PI and flesh out a more thorough and detailed plan.

There are three main principles that I try to work into any research project:

  1. Try to think of a design that can offer you more than one way to address your point.

  2. Think about minimizing measurement noise.

  3. Show all your final work, preferably publicly.

I’ll elaborate on the first two points now. There is so much to say about the third point that there is a whole chapter about it later.

2.2 Planning for converging evidence

There are many ways to write an adequate scientific article. Often, scientific articles consist of only one experiment, analyzing one dependent variable. Sometimes a report this simple is striking and clean, and all you wanted to know about that topic. However, usually such a report leaves a lot of unanswered questions. In the best-case versions, the single-experiment paper reports a beautifully designed study, with obvious predictions, clear results, with results coming from multiple dependent measures and all converging on the same answer. I don’t believe that these lovely, succinct papers come from single-shot attempts. I think they come from extensive piloting, and eventually the judicious decision to write a brief paper about the ultimate design rather than write a long paper including every step that led to the ultimate design. While we will never hide the preliminary work that went into a published paper, the ultimate communication we make will reflect the evidence that we are confident about: experiments conducted after piloting, when we are sure we understand how the task is working. Though you should not assume that all the data you collect will be publishable quality, or that you will write a paper about each experiment you conduct, you will save yourself some effort though if you consider in advance what kinds of evidence might help you make a converging case, and how much opportunity to acquire that evidence you can work into your experimental designs.

One important tip is to always measure and record the rawest dependent variables possible. For example, we are always measuring accuracy in our work. With accuracy, you can record whether a given response was correct or incorrect. That might be all you are planning to analyze, but it limits you. It might be interesting to know in what way a wrong response was wrong. You might not realize it at the start of your project, you might later think of a way to use errors to determine between two otherwise vague possibilities. By recording only “right” or “wrong”, you are losing information that might have been useful.

I’ve regretted decisions about this myself. I have a few data sets measuring spatial serial order reconstruction via mouse click responses. My program recorded a lot, but I did not manage to get it to record response times for every mouse click. There are exquisite analyses that can be done with this information (e.g., Chevalier, James, Wiebe, Nelson, & Espy, 2014) that I unfortunately cannot perform on those data sets (e.g., Morey & Miron, 2016; Morey, Mareva, Lelonkiewicz, & Chevalier, 2018).

Our resources are always limited: there is never as much money, or time, or participant energy as we could spend. Make the most of your resources and record as much as you can about your participants’ response. Sometimes it will take extra effort in programming your experiment, but if you get to N=40 and then decide you want those data, you will not only have to figure it out, but you will have to begin collecting data again. Figure it out first, or at least weigh the risk of recording less with the PI before beginning data collection.

Sometimes, convergence of evidence will only be possible with multiple experiments. Assuming that we have the resources, we want to write one very convincing paper rather than several papers with evidence that doesn’t support a complete argument. This also lets you better control your narrative; writing about one isolated and interesting result, you will need to discuss the myriad possibilities for interpretation that you have not yet ruled out. Putting multiple lines of evidence in one paper means that you can rule out more interpretations, which strengthens the interpretation your data support.

It is unlikely that you will be running an experiment, then publishing it, and repeating that process, so you want to plan for serial data collection, for efficiently processing results, for being ready to collect data during convenient seasons for participants, etc. if you want to write convincing papers that are bursting with clear evidence.

2.3 Minimizing noise

Another factor that separates the so-so evidence from the convincing evidence is how clear and obvious the effect looks. Of course we will be doing inferential statistics, but ideally these only confirm what is already obvious in your plots. This means you should collect more data than you think you need to detect an effect of a particular size (more on that later). You should also think carefully about all of the irrelevant factors that could affect your participants’ responses and do what you can to neutralize them in your experiment’s design and protocol. These are things like interruptions (phones beeping in the middle of a task, fire alarms sounding), irritating background noises, fatigue, contingencies that unintentionally reinforce random responding, or individual differences beyond those we normally expect (e.g., participants under the influence of a drug). Much of our lab’s standard procedures have been adopted through experience and practice to minimize extraneous measurement error. Adapting your protocol from an existing one helps you benefit from someone else’s experiences.

2.4 Illustrations of scope and output of student projects

It can be difficult to imagine what a reasonable project looks like if you haven’t done one. It is also difficult to generalize enough to say what a great UG dissertation or PG thesis looks like. I’ll try though. Naturally every project is custom, but I’ve attempted to illustrate a holotype - the modal characteristics of a solid, successful project - for UG and PG projects.

2.4.1 Holotype: UG dissertation timeline and workload

In the UG dissertation scheme at Cardiff University there are milestones built into the system. You must have a working title and abstract after about 3 weeks. There is little time for thinking about what research question you are asking and how you will answer it, because by the end of October you need to be thinking about how you will implement your design.

Start off in week 1 (or earlier if it suits everyone) by meeting with your PI and informally chatting about your interests and what skills you hope to learn during the project. Your PI will have a mental short list of possibilities and can quickly see which ones are most likely to suit you, and how they might be adapted to suit you better. You should leave this meeting with a few ideas and a list of readings.

Your project will be one experiment with at least 30 participants. If you are working collaboratively with other students, you might be running a series together or a larger, more complex design that requires a larger sample. But if you are working independently, count on only having time and resources for a single experiment, with ~30-60 participants (depending on session length).

Once you have settled on a question, find your deadline, work backwards, estimate how much time you want in between major milestones (e.g., getting feedback from your presentation and submitting, writing after finishing your analyses, analyzing your data after collecting them, etc.). You will probably find that you want to have at least begun data collection before winter break. This is also a bit early in terms of when most UG students collect data, so you will have plenty of participants.

This gives you one month to implement your experimental design and develop your protocol. Here is where your PI can help. There will be lots of ready-programmed experiments available from other lab projects. Your PI can help you limit your choice of paradigm to one that the lab already uses in some form, identify modifications you may need to make, and help you see how these modifications fit with what has been done previously. You will learn a little about programming, but unless you already know how, it will be extremely hard to learn to program and create a useful program from scratch in a month. Before collecting data, you will have already:

  1. Written an experimental protocol explaining step-by-step how to run a session
  2. Tested at least one pilot participant to make sure your protocol includes every step needed
  3. Gone over the output from your program with one of the senior researchers to make sure it records everything you need to answer your research questions
  4. Scheduled a real participant for a time when one of the senior researchers in the lab can supervise and check that your implementation of the protocol is fine
  5. Created an Open Science Framework page for your experiment (with the PI as a full-fledged user), uploaded your experimental software, your protocol, paperwork, and a written description of your method and planned analyses

When you return from winter break, you will finish your data collection and begin processing your data. Here is another moment to seek advice from the PI. Our lab manual codifies many useful procedures for processing data, but they require understanding R scripts well enough to copy and modify existing scripts. Your supervisor can help you with this, and direct you to ways to learn how to get specific things you need from the data. You should complete our standard anonymizing steps on your compiled raw data (either on your own or with help from a senior lab member), and then save the resulting data file to your Open Science Framework page. You may now analyze the data and prepare for your March lab meeting presentation, where you will receive feedback on your analyses. After that, you have a few weeks to finish writing.

2.4.2 Holotype: PhD thesis timeline and workload

The PhD thesis is a substantial body of work carried out over three years. A good series of PhD studies should produce at least one substantial journal article. Usually, a single “chapter” of the PhD includes the contents of one published or publishable paper. Such a chapter may contain a single experiment or a series of related experiments. A good PhD thesis in cognitive psychology typically contains at least three empirical chapters, plus an introduction chapter and a concluding chapter.

A PhD thesis must be cohesive. It cannot include chapters about different things, because the introduction and concluding chapters must make clear how the chapters fit together. So, you must stick closely to whatever topic you and your PI agree on when you begin.

It is a good idea to identify tasks that you can complete in parallel so that you are not reactively collecting data, seeing how it went, deciding what to do, and repeating the process. I think it is ideal to jump-start the PhD project by simultaneously working on a related lab project while designing your own first project. This gives you hands-on experience with the lab procedures you will be using so that you can start your own project quickly when it is ready, and potentially involves you in one lab publication besides any that come from your PhD experiments. An ideal timeline could involve starting with related data collection upon arrival, discussing a way to meta-analyze your topic with your PI and working on that alongside data collection, and then using the knowledge acquired from meta-analysis to plan your first experiments. You want to be ready to collect some data in the spring, but to plan for the bulk of your data collection to occur in the academic Year 2 and the fall of Year 3, when participants are widely available. Ideally, January - September of Year 3 is for writing, knowing that you could probably collect another experiment in the spring semester if you think it would help.

Each empirical chapter should be a self-contained empirical paper (that perhaps has been submitted for publication). You will have been preparing these gradually during Year 2 and the beginning of Year 3, so some of the months reserved for finishing writing will be spent implementing feedback from peer review. Ideally you will also have some time to consider preparing applications for post-doctoral jobs.

2.5 Making the most of your time

Do those timelines sound reasonable? Experience has taught me that there is somehow never enough time. Every hour you can save by relying on existing procedures and pipelines will help you make the most of your effort. You will also naturally find ways to innovate our pipeline as you work, so as others have paid it forward to you by programming experiments you can modify and figuring out useful R functions and scripting processes that you may copy, you will pay it forward by periodically incorporating something you have discovered into our process.

2.6 Workload expectations

There is no getting around the fact that scholarship is competitive, and that no matter how organized and smart you are, you will need to work very hard (and almost certainly sometimes for longer hours than preferred or strictly required) to complete your marked work to the highest standard. Especially at the post-graduate level, the quality of your work is your ticket to your next job. The better your work is, the more options you will have in your career after university. I do not mention this to frighten you into over-working, but rather to encourage you to think about how you want to manage this. There must be a balance between work and rest so that when you are working, you are able to make the most of your time.

Students are expected to treat their studies like a full-time job (e.g., 35-40 working hours per week, with holidays to be taken at convenient times with respect to the work). I encourage every new starter to speak with me about this: I need to know what your plans and habits are, and I will accommodate them to the extent possible. Because I do not believe that everyone works best on the same schedule, I encourage every individual to think about the ideal hours they would like to maintain, and whether they prefer a strict or flexible schedule. If your ideal hours are very unusual or somehow incompatible with the needs of your project, we need to discuss that as soon as possible. In all projects there will undoubtedly be times when hustling is needed, but fortunately these times are usually predictable (e.g., to get data collection started in time for recruiting free student participants, preparing developmental studies to be run during half-term breaks, submitting an abstract for a conference deadline, etc.). Everyone should be aware that when they have worked long hours, they are entitled to take some extra time to rest and recover.

My hours and habits should not be taken as directives by trainees. Though I try to keep weekends for myself and to work 7-8 hours per day, I often work more than that and sometimes work strange hours when I am inspired to or trying to meet a deadline. I feel great relief to get something off my desk, so I may well send an email after typical work hours are over. I’m often traveling to attend conferences or visit other universities, which may be in different time zones (another reason you may get a message from me at an odd time).

I shut my work email off when I’m not working and sometimes also when I am concentrating on writing. I do not receive any auto-notifications to my personal devices about email or Teams messages, ever. I advise everyone to do the same. My assumption is that you will receive messages at some time on every work day when you have explicitly planned to attend to them, not that you will be always on-call. You should not imagine that receiving an email from me means that I expect you to instantly answer it. You are never obliged to answer me at night or on weekends even if you receive something from me after hours.