Sunday, December 15, 2013

A guide to post publication peer review

Interesting things are happening in science publishing. In the recent months we witnessed how multiple post publication peer review platforms were born and gained popularity. In this post I will try to clarify what is post publication peer review, what are the differences between existing platforms and what those changes mean for the future of publishing. But before I begin let me say a few words about how most publishing works right now.

We were always joking that we should start a new journal that would accept everything and use post publication peer review. To give it enough gravitas to compete with Nature and Science we decided to call it Truth. In the  picture (on the right) - a long time supporter of Truth - +Jonathan Smallwood .

Peer review before the intertubes

Say you are a scientist (if you are reading this it is very likely that you are, but let me try to make this topic understandable for people outside of academia). You have just discovered some exciting properties of a chemical molecule, or came up with a new theory of attention. You would like to share your findings with everyone else. You write a paper and submit it to a journal of your choice. The journal assigns an editor who tries to find two or three reviewers that read your work and look for methodological mistakes (statistics, experimental procedures) and flaws of reasoning. They can suggest changes, additional experiments. When/if they are happy with your manuscript it’s gets accepted, you open a bottle of sparkly and celebrate. Otherwise the paper gets rejected. This usually means that you will have to substantially change it or try again with another journal and another set of reviewers. The purpose of this system is quality control. Only “good” science should be able to pass the test of peer review. However there are some problems with this approach:
  1. The acceptance of a paper depends to a great extent on luck. Depending who are your reviewers your work will be assessed differently. It is not unheard of that papers get rejected due to reviewers working under tight deadline and outside of their strict expertise. This of course can work both ways (poor papers being accepted, good papers being rejected).
  2. The content of reviews is not available publicly.
  3. After publication there is no easy way to voice your opinion about the the quality of a paper or potential problems with the described science.
  4. In most cases reviewers remain anonymous and get no credit for doing the reviews.

To put it in more relatable terms: imagine that Amazon would only sell books selected by their internal reviewers. There would not be any way of posting your ratings or opinions about the books nor to see what internal Amazon reviewers thought about them. Total blackout.

Post publication peer review to the rescue

A simple solution to this problem is to provide a platform for public reviewing of academic papers after their publication. There is nothing fancy about this idea. It is basically what IMDB and Amazon have been doing for ages. There are, however, some issues that make academic papers different from films and books. Below I will describe three in my opinion most prominent post publication peer review platforms.

PubPeer.com

PubPeer.com is currently probably the most popular platform for post publication peer review. It has been featured in several media outlets and reviews posted on it lead to several retractions. PubPeer is very simple to use: you just need to put a DOI of a paper, register an account (you need to be either first or last author on an existing paper to do this) and you are ready to review. The platform praises anonymity as a necessary mechanism for unbiased reviews and this is the default way of posting reviews (although no one prevents you from signing your review). The anonymity (and lack of consequences towards researchers reputation/carrier) is probably the reason why PubPeer is so popular. Unfortunately this also means that 90% of the reviews are negative. If anyone bothers to write a review they do it just to point out flaws and criticize papers.

PubPeer is hosted by anonymous (who would have guessed…) group of people and is funded through their private resources. The authors have turned down donation offers in the past to avoid any accusations of conflict of interest.

Publons.com

“A Publon is a fundamental unit of academic publishing” we can read on their website. The idea behind the website is 180 degrees different from PubPeer. What Publons is focusing is giving reviewers credit for their work. Reviews are therefore sign with the real name of their author and if they get voted up (or endorsed) by the community they receive a DOI and can be cited as publications. Publons are also allowing you to submit the reviews you did for a journal (pre publication reviews). You don’t even have to make them public - Publons promises to verify your review with the journal to give you credit anyway. Reviews are (or very soon will be) indexed by Google Scholar which should increase their impact.

Publons is a startup run by some very enthusiastic and dedicated people from New Zealand. Their business plan includes helping journals finding the right reviewers, by building a platform for pre and post publication reviews. As with all startups it’s future is unclear. I would not be shocked if when it gains momentum it will be bought by one of the big publishers (the same thing happened to Mendeley and Frontiers).

PubMed Commons

PubMed is an NHI run library of biomedical literature indexing publications from many, many journals. Recently they started a closed (invitation only - get in touch if you want one) trial for putting comments under the abstracts of the papers they index. Comments have to be signed by your own name (again - verified through your publications), and essentially are nothing else than reviews. There are two advantages of PubMed Commons: i) your reviews will appear on the website visited by thousands of researchers every day, ii) the platform is run supported by US government money. There is one caveat though - PubMed is only for biomedical research. Sorry physics!

On the horizon

An honourable mention goes to Libre. It’s yet another post publication peer review platform, that is still under construction. Little is known about their policies and business model, but they published a super cute video describing the ideas behind it:

Too many cooks?

One may think that we don’t need so many platforms, that they only cause confusion. I disagree. I think it’s great that we have competition. It’s very important, especially in this early stage, to try different ideas. Some people will want to get credit for their reviews, some will prefer to be anonymous. The only problematic aspect is how to aggregate reviews from all platforms. This should not be a problem if all of the platforms provide publicly available APIs. Publons has one, PubMed Commons is working on one. Unfortunately, PubPeer, even though it has an API, gives access to it only to selected partners. In their view having multiple platforms is a bad thing, and they are willing to use the control over the access to their API to fight the competition. When I approached them with the idea of a review aggregator they denied me access to their API. Let’s hope they will change their minds in the future and open their API. That is what open science is really about.

So which one to choose?

Considering available options the decision which platform to use is not that hard. If you want your review to be anonymous use PubPeer. If you are not afraid to put your name under your review and you would like to get academic credit for it go for Publons. If you see the appeal of the impact that PubMed exposure gives go with the Commons.

The bigger picture: the future of publishing

Post publication peer review is just the beginning of the revolution. Soon the “golden” days of researcher giving and curating their papers for free to commercial publishers so they could sell them to other researchers will be gone. Papers will not have to go through random and arbitrary peer review behind closed doors. Researchers will self publish their work (see excellent blog post by +Micah Allen about self publishing) and rely on post publication peer review. Welcome to the brave new world!

Wednesday, August 14, 2013

What "Hitchhiker's Guide to the Galaxy" is really about

I was on holiday recently lying on a meadow somewhere near Furka pass in Switzerland. It was the middle of the night and the sky was cloudless. Far from any civilization I could see the night sky in all its glory - with all the tiny stars and the clear shape of the Milky Way. Truly stunning view!



Coincidentally I was also reading Douglas Adams "Hitchhiker's Guide to the Galaxy". It's brilliant! Funny and smart. It struck me what it really is about. Let me give you a hint by quoting a paragraph:

"Han Wavel is a world which consists largely of fabulous ultra-luxury hotels and casinos, all of which have been formed by the natural erosion of wind and rain. The chances of this happening are more or less one to infinity against. Little is known of how this came about because none of the geophysicists, probability statisticians, meteoranalysts or bizzarrologists who are so keen to research it can afford to stay there."

Yes! It is basically saying that because the universe is so vast, the probability of extreme events (such as spontaneous formation of luxury hotels) happening by pure chance is not that low. This is nothing else than the problem of family of statistical test and multiple comparison correction. The longer you draw samples from a random distribution the more likely it is that you will find one very extreme value!

Of course I cannot fool anyone that H2G2 is about multiple comparison correction. Obviously it's about life, the universe and everything. It is just my personal bias. Being obsessed about a certain problem I'm more likely to find patterns related to it. This is, however, a topic for a whole new post :)

Wednesday, May 1, 2013

Dreamitts

I live in a culture of success. I'm being constantly told that if I put my mind to it I can achieve anything. "Yes we can!" It's great - it gives me motivation when I need it. When things get tough I am reminded that if I truly believe in my dream I can move mountains. I would not be where I am now without this push. It's awesome, but not for all dreams.

I call those other dreams dreamitts. They are bad dreams, but unlike nightmares, they will not scare you away. Quite the opposite, they are like bright light that lures you in. It's a candle shining in the dark and you are the hopeless moth - when you get too close you will just burn. Dreamitts will make you suffer while trying to achieve them. You will fall. A lot. You will hear "You can do it!" stand up just to fall again. And since dreamitts are so unattainable they seem much more valuable than they really are.

Our lives are short, our energy is limited. Some dreams are just not worth pursuing. Sometimes it is just better to let it go: to admit it's too hard and too painful and to channel your creativity, passion and energy into something that is actually worth it.

But don't think about it as a failure. It's a terribly difficult decision to make - even if painful, status quo is less risky than a change. I would not call it giving up either: it is putting your awesome potential into something better suited.

Wednesday, April 24, 2013

You are your brain

It is a puzzling mental exercise to reflect upon the processes behind your own thoughts. Philosophers have been struggling with the problem of consciousness for ages and recently entered into dialog with neuroscientists to look for its neural correlates. The road to understanding how the human brain works is very long and we have only just started this journey. Despite efforts of many great minds, our knowledge of mechanisms underlying human activities such as  love, compassion, problem solving, speech, or even vision is still far from complete. Undeniably we have made some progress: we have learned how to map different brain areas to minimize damage during brain surgeries, treat depression and Parkinson’s disease by directly stimulating a particular part of the brain and develop drugs that alter chemicals in the brain and allow to manage some mental disorders.
Despite the limitations of our knowledge one thing is certain: whoever we are, whatever we do, and whatever we feel originates in our brains. In fact we are our brains - they define us. It should not be a surprise that more and more well known behavioural phenomena are being mapped in the brain. After all there is nothing more to what we are: there is no little man hiding in our skulls watching all the sensory inputs and pressing buttons to make our limbs move. There is no meaning in saying: “My brain made me do it” since your brain makes you do everything. Therefore talking about you and your brain as two distinct entities has little sense since they are indistinguishable. Of course the brain can be influenced by many factors such as experience, drugs and illness, but it is still you that changes not a one some organ separate from a magical box responsible for free will and consciousness.


With more sophisticated tools: stronger scanners, better mathematical models, and deeper knowledge of molecular mechanisms we will understand more and more about the brain. In some time we will be able to explain how biochemical processes translate to complex behaviour such as emotions. We put so much effort and resources in neuroscience because
the human brain might be the most important mystery yet to be solved. Maybe even more important than life on Mars or new species living in the depths of the oceans (if we really have to pick one). Nonetheless, disentangling the intricacies of the human brain should not frighten you. Quite the opposite - there is beauty in any complex phenomena and joy in trying to figure out its components. At the end of the day, understanding the complicated mechanism behind emotions will not make your feelings less genuine. So even though, you are your brain – around 1300 grams of greyish tissue - it does not make you in any way less special.



I have written this wee piece for an art project that will soon be displayed in Berlin (I'll keep you posted!). Many thanks to +Marek Kaluba , +Samyogita Hardikar , and +Daniel Margulies for helpful comments.

Sunday, March 10, 2013

The tempting illusion of simplicity

"Plurality is not to be posited without necessity" says one of the major rules of scientific thinking - The Occam's Razor. In other words when we have two explanations of the same evidence the one that is less complicated (or require fewer assumptions) is more likely to be true. Of course it is rarely the case that the models we are considering are explaining the data to the exact same extent, but modern statistical techniques have been developed deal with this (see BIC, AIC and Bayes factor). In short, those methods combine evidence for each model (such as goodness of fit or likelihood) with complexity of the said model (for example the number of free parameters) in a way that penalizes overly complex solutions. 

All of this should be nothing new to anyone dealing with models, data, and theories. There is, however, one additional aspect concerning the complexity of explanations - a social one. We intrinsically like simple stories. It's not only because it is easier to get our heads around them, but there is a natural drive to clean and simple explanations. We love overarching and unifying theories (like the Holy Grail of physics The Unified Field Theory or the The Free Energy in the brain). We also prefer simpler methods and models. That is partially why massive univariate modeling is still more popular way of analyzing data than much more sophisticated multivariate models. It's just much easier to interpret! 

What is more the reviewers and editors of our papers like simpler stories. Many papers presented in high impact journals are telling powerful but simple stories. The problem with this bias is that many natural phenomena (such as the human brain not to look too far) are unlikely to be explained by simple mechanisms. However, we still take the existing data and bend the interpretations to end up with a simple and attractive stories. And don't get me wrong it is not because we want to please our reviewers and readers (well - I hope it is not only always that), but simpler theories are easier to work with. It is our own mental capacity that limits the theories we are going to work with. However, the true answers might be much more complex than we would like them to be.


Sunday, February 3, 2013

Making data sharing count

Consider a typical fMRI study: 
  • Twenty participants scanned for an hour = 10000 USD.
  • Research Assistant to run participants = 20000 USD.
  • Postdoc to invent the study and write it up = 40000 USD.
70000 USD later science is richer by an eight page paper, peer reviewed and published in an academic journal. The authors might look at the data again some time later, maybe join it with some other of their dataset to improve power. Maybe. Or maybe they will not have time. We may never learn if there was anything more in the data (all 360 million datapoints of it) than what those eight pages described.

Most scientists agree that sharing data makes sense and leads to better, more reproducible, transparent, and objective science. Funding agencies (the guys who turn your taxes into academic papers) understand how expensive data collection is and want to squeeze as much as possible out of existing data. But the perspective of an individual scientist is different. Sharing data does not come for free. You need to clean the data and describe it properly so other could make good use of it. You also risk that someone will try and fail to replicate your findings - unearthing a mistake in your analysis. All that for what? So someone else could take YOUR data find something interesting that you have missed and publish it? Leaving you with no credit for the data collection, nothing to put on your CV when you are going to face the tenure track committee?

Luckily not all scientists think this way, but plenty do. Even though there are many visionaries and idealists in science (luckily!) in many situations it is a dog eat dog, you publish or you perish dynamic. I don't believe this is fundamentally wrong - competition is driving development. Besides entities distributing money in science have to somehow make their decisions. Therefore we should not fight this, but try to tap into the existing system of academic credit.

Together with Daniel and Mike we have recently written a paper describing an attempt to increase the motivation of an individual researcher to share data. Instead of just putting your data on a website and not getting anything in return one would write a short paper describing in details how the data was acquired and how it is organized. Such data paper is publication like every other paper. It has a DOI, can be cited, and has to be peer reviewed before being accepted. This simple idea solves multiple problems:
  • Through citation data producers get appropriate credit. Interesting data sets will lead to highly cited papers.
  • Peer review process assures that the quality of the data and metadata leans to trouble free reuse.
  • A separate publication allows more space for detailed description of acquisition methods in contrast to just a few paragraphs of a typical cognitive neuroscience paper.
  • All people involved in the data collection (including research and lab assistants) can co-author the paper without concerns of "dilution of credit".
By no means this is a new idea: it has been implemented in other fields (see our paper for more details). It just needs to gain momentum (and this is the main reason for this shameless plug ;). There are already several neuroimaging journals that will accept data papers: GigaScience (they will also host your data), Neuroinformatics and Frontiers in Brain Imaging Methods. There is really not much to loose. With little effort you can get a publication, promote and share your data. So what are you waiting for? Publish a data paper to increase the impact of your research and receive credit for your data sharing efforts!

Sunday, January 27, 2013

"The Shallows: What the Internet is Doing to Our Brains" - Book Review

I'm impatient by nature. When I'm on the interner, however, I'm like Twitchy on coffee (now imagine what happens when I'm browsing the web after having a cup of the dark brew). I don't stay on one website longer than five minutes, I struggle to finish watching a youtube video without checking something in the background, I catch myself mindlessly going to websites and asking myself "Why did I opened this tab?". It's bad. It's a constant crave for new information, new stimuli, something surprising, something funny.

But it didn't use to be this way. I used to be able to finish reading a book in one sitting (even though it did not happen often). Now I struggle with one academic paper without a break. Something changed in me about how hard it is for me to focus on one task for an extended period of time. The obvious thing to blame was the Internet. And I know I'm not the only one. Why services like rescuetime.com and keepmeout.com are so popular? Why when I mention pomodoro you think about productivity instead of pasta sauce?

I turned to research for an answers, but I was too lazy to do a proper literature review myself. I needed something to start. That's how I stumbled across "The Shallows: What the Internet is Doing to Our Brains" by Nicholas Carr. The book starts very slowly with history of neuroscience, computers, and books. Then when it get into the meat of the problem it suddenly ends. The introductory part is interesting (especially if you did not have much to do with those topics), but some seem like a space filler that is forcefully related to the real topic. Nonetheless, I have learned a handfull of interesting little facts. For example did you know that Freud, before he started his psychiatric practice was working in the field of molecular neuroanatomy? That was as close to neurophysiology as one could get those days! Another fun fact concerned Nietzsche: when his sight deteriorated he started using one of the first type writers and soon became a proficient touch typist (which apparently changed his style of writing).

But apart from those little gems there are other bigger ideas laid on the pages of "The Shallows". First of all this is not he first time a technological revolution has changed how we exchange and consume information. The invention and popularization of printed books lead to wide spread (and for obvious reasons undocumented) changes of our cognitive capabilities. People were able to acquire knowledge no only through personal interaction but from then on also by spending some quality time with a book. So maybe the skill of focusing on a read that I am missing now and that seems so natural wasn't so natural back in the days.

This leads to another interesting thought. Maybe evolution shaped us in a way that promotes multitasking and rapid switches of attention. After all, those seem like very useful and practical skills in an environment full of predators. So maybe the reason why the Internet has such a strong effect on us is because it taps into intrinsically natural set of skills. I know that comparing watching YouTube while checking your Facebook, to peeling fruits while looking out for lions seems far fetched, but there might be something to it.

Unfortunately the book provided only a handful of useful references to directly related studies (which I'll try to review later), but maybe this accurately describes the state of our knowledge. We still have not figured out how human vision or memory works, so maybe we should not expect much from research on a complex interaction between culture, technology and attention (although crazier things have been attempted using fMRI). Besides if it was a proper literature review not a pop sci book it would probably not get shortlisted for The Pulitzer Prize ;)

PS A friend reviewed this book for the Science magazine - please have a look. If you have access... Cannot wait for open access to become a standard!

PPS I'm done with the book and since I hate accumulating things (makes moving really hard) I'm happy to give it (mail it if needs be) to anyone who gets in touch with me first.

Sunday, January 20, 2013

On how we estimate value

A friend of mine once had a crush on a girl. The girl wasn't really interested, but valued him as a friend and was too polite to tell him he does not stand a chance (or maybe she just enjoyed the extra attention - who knows!). So the chase went on reaching pretty pathetic levels. At the same time another girl was basically throwing herself at him. Yet he wasn't paying attention to her and preferred to chase an illusion.

It's not a single case. Even if you have not experienced it yourself (on which I congratulate you - I'll try to touch on individual differences later on), you must've heard similar stories from your friends. Actually if you look for most popular dating advice you will learn that the trick is to let go and maintain the magical balance of not caring and being interested (or as John Green would say in his witty way: "dumpees should fight the clingy urge"). Speaking more general my friend was assessing a potential relationship mostly basing on the fact how hard it would be to get. Other more objective criteria (such as for example disturbing lack of sense of humour of the aforementioned femme fatale) did not matter that much.

But this does not happen just in the field of romance. We tend to apply a similar rule to jobs and positions. A highly competitive position will attract even more people. After all if so many people want it it must be good! Some people reapply for some jobs multiple times until they get it often discarding posts which would be a better fit. You could observe this in Google recruitment for engineering positions. Thousands of applicants competing for a few positions, at least seven stages of interviews, high rejections rates, and admitting openly they reject many good candidates - yet many peoples try up to five times (and probably a couple of years) to get a position. At the same time many of them would be equally happy at a startup company somewhere in California.

The same rule applies to physical possessions and services. A at the beginning of the month I accidentally took the wrong train and ended up spending the night at various train stations trying to get to my final destination. I started talking to a guy who also was in a similar situation. He run a barber shop in a small town. He told me what was the "secret" to the success of his business. There were many barber shops in this town. Most of them provided services on a similar or indistinguishable level. His shop, however, charged more than others providing an illusion of a premium service. Despite the higher price and similar quality they were always fully booked. And as in the examples above the clients used the price (more expensive == harder to get) to estimate the quality (value). Of course this trick is nothing new and is known in economics as premium pricing.

What I'm really interested in is what science has to say about this phenomenon. The Nobel prize winner Daniel Kahneman had several theories on how probabilities and absolute values of events translate to their utilities. His Prospect Theory gives a mathematical model explaining how we overweight extreme events (those that are highly unlikely) disregarding their true value (of course if I understand it correctly). The problem is that we don't know the true value (neither the probability) and we have to estimate it. Search for scientific work on this topic has proven to be hard mostly due to the fact that it must exist across fields under different names. There is at least one study showing that the "playing hard to get" tactic in romance is popular (surprise, surprise...). If you have any hints where should I look please let me know!

Last but not least I am not claiming that this heuristic is always bad. After all we don't have access to objective value (if something like this even exists). Estimating it based on the fact how much effort we need to put into trying to get it might be in many situations the best heuristic. Maybe a more expensive hairdresser is in fact better, maybe a position that is harder to get would be indeed more fulfilling, and the girl that my friend was chasing so relentlessly would be more likely o be a keeper.

Sunday, January 13, 2013

A more probabilistic view on multiple comparisons problem

Even though this blog is not going to be only about multiple comparisons (I could not think of another name), I decided to write about an old problem in slightly new way.

Multiple Comparisons

Whenever we are testing many hypotheses and are trying to figure out which of them are true we stumble upon so called Multiple Comparisons problem. This is especially evident in fields where we do tens of thousands tests (such as neuroimaging or genetics). So what is the big deal? Imagine that you divide the brain into a buch of regions (voxels) and for each of them you will perform some statistical test (checking for example if this part of the brain is involved in perception of kittens). Some of the regions will yield high statistical values (suggesting relation to kittens) and some will not. Lets try to show this with a simple simullation.

Let's assume for now that we will test 100 voxels and only 10 of them will be related to kittens. We will model both populations of voxels using Gaussian distributions. Noise distribution will be centred on zero opposed to signal centred on three.

In [2]:
import numpy as np
noise_voxels = np.random.normal(size=90, loc=0.0, scale=1.0)
signal_voxels = np.random.normal(size=10, loc=3.0, scale=1.0)

Lets plot this

In [6]:
import pylab as plt
figsize(10,6)
plt.hist([noise_voxels, signal_voxels], bins=20, label=['noise', 'kittens'], histtype='step', fill=True, stacked=True)
plt.legend()
Out[6]:
<matplotlib.legend.Legend at 0x105eb9790>

Even though noise is dominating in this example it would be very easy to draw a line distinguishing non-kitten related voxels from those that really do say "meow". What does it has to do with multiple comparisons will be clear in a moment.

Firstly let's show that this is just a simulation and depending on what mood my computer is in the results will be different. Here are four instances.

In [7]:
for i in range(4):
    plt.subplot(2,2,i)
    noise_voxels = np.random.normal(size=90, loc=0.0, scale=1.0)
    signal_voxels = np.random.normal(size=10, loc=3.0, scale=1.0)
    plt.hist([noise_voxels, signal_voxels], bins=20, label=['noise', 'kittens'], histtype='step', fill=True, stacked=True)
    plt.legend()

We can operate on the theoretical distributions instead of just the simulations. Since we are dealing with two Gaussians let's plot two Gaussians.

In [12]:
x_range = np.linspace(-3,6,100)
noise_samples = 90.0
signal_samples = 10.0
snr = signal_samples/noise_samples
from scipy.stats import norm
plt.plot(x_range, norm.pdf(x_range)*(1-snr), 'b', label="noise")
plt.plot(x_range, norm.pdf(x_range,loc=3)*(snr), 'g', label="kittens")
plt.legend()
Out[12]:
<matplotlib.legend.Legend at 0x105b7ac50>

Now we can clearly see that the overlap between the two distributions is fairly small. Notice that there are two important parameters that can influence this: Signal to Noise Ration (SNR) and location of the signal distribution (also known as the effect size).

The multiple comparisons problem is all about... well multiple comparisons so in other words the number of tests we make. In our example this is equivalent to how many voxels we have). So let's show this by upsampling our data! Let's say we will be able to divide each old (big) voxel into eight small voxels.

In [13]:
noise_samples = 90.0*8
signal_samples = 10.0*8
snr = signal_samples/noise_samples
plt.plot(x_range, norm.pdf(x_range)*(1-snr), 'b', label="noise")
plt.plot(x_range, norm.pdf(x_range,loc=3)*(snr), 'g', label="kittens")
plt.legend()
Out[13]:
<matplotlib.legend.Legend at 0x10756a150>

Surprisingly nothing has changed... But we have more voxels and did more comparisons (tested more hypotheses)! True, but becaue we only upsampled the data we just created identical copies of old values. SNR thus stayed the same. However, things change when we consider a more realistic situation than "10% of the brain selectively responds to young cats". Out of 60000 voxels (average head size, 4x4x4mm resolution, skull stripped) 100 will respond to kittens.

In [77]:
noise_samples = 60000.
signal_samples = 100.
snr = signal_samples/noise_samples
plt.plot(x_range, norm.pdf(x_range)*(1-snr), 'b', label="noise")
plt.plot(x_range, norm.pdf(x_range,loc=3)*(snr), 'g', label="kittens")
plt.legend()
Out[77]:
<matplotlib.legend.Legend at 0x1093126d0>

Where are the cats gone?!? Let's have a closer look.

In [78]:
plt.plot(x_range, norm.pdf(x_range)*(1-snr), 'b', label="noise")
plt.plot(x_range, norm.pdf(x_range,loc=3)*(snr), 'g', label="kittens")
plt.legend()
plt.xlim([0,6])
plt.ylim([0.00,0.01])
Out[78]:
(0.0, 0.01)

Haha! If we zoom in we will be able to find the signal distribution dwarfed by the noise. The problem is not the number of comparison we do but the fraction of those comparison that will be yield no signal. If you look carefully you will notice that the crossing point between the distributions increased with decreased SNR. This crossing is a potential candidate for a threshold. Let's try to find this point.

In [85]:
from scipy.optimize import fsolve
fsolve(lambda x : norm.pdf(x)*(1-snr) - norm.pdf(x, loc=3)*(snr),2.0)
Out[85]:
array([ 3.26443494])

The interesting aspect is the relation between this crossing point and SNR.

In [80]:
snrs = np.linspace (0.3,0.005, 1000)
crossing_points = []
for snr in snrs:
    crossing_point = fsolve(lambda x : norm.pdf(x)*(1-snr) - norm.pdf(x, loc=3)*(snr),2.0)
    crossing_points.append(crossing_point)

plt.plot(snrs, crossing_points)
plt.xlabel("SNR")
plt.ylabel("crossing point")
Out[80]:
<matplotlib.text.Text at 0x109629110>

As we can see it reises sharply for very small SNR values. Another popular option for picking a threshold is controling for False Discovery Rate. The fraction of false discoveries among all voxels labeled as significant. This is equivalent to the ratio of area under the blue curve right of the threshold to the sum of the areas under the blue and green curves right of the threshold. This areas are summarized by the Cumulative Distribution Functions (CDFs).

In [87]:
thr = 3.26
(1-norm.cdf(thr))*(1-snr)/((1-norm.cdf(thr))*(1-snr) + (1-norm.cdf(thr, loc=3))*snr)
Out[87]:
0.45640041955468291

Another important value is the percentage of missed voxels.

In [89]:
norm.cdf(thr, loc=3)
Out[89]:
0.6025681132017604

As mentioned before a popular option in dealing with Multiple Comparisons is to keep the FDR at a certain level (usually 0.05). Let's see what happens to percentage of missed voxels if we do this at different SNRs.

In [83]:
missed_voxels = []
fdr_thresholds = []
for snr in snrs:
    fdr_thr = fsolve(lambda x : (1-norm.cdf(x))*(1-snr)/((1-norm.cdf(x))*(1-snr) + (1-norm.cdf(x, loc=3))*snr)-0.05,2.0)
    missed_voxels.append(norm.cdf(fdr_thr, loc=3))
    fdr_thresholds.append(fdr_thr)

plt.plot(snrs, missed_voxels)
plt.xlabel("SNR")
plt.ylabel("percentage of missed voxels")
plt.figure()
plt.plot(snrs, fdr_thresholds)
plt.xlabel("SNR")
plt.ylabel("FDR corrected threshold")
Out[83]:
<matplotlib.text.Text at 0x10966df90>

From this plot we can see that when we decrease SNR, even though we control for FDR we are missing a lot of voxels. For extremely low SNR and small absolute number of signal voxels chances of finding anything are very slim.

Take home message

In this inaugral post I was trying to show multiple comparison problem in a slightly different view. I hope that from those simple simulations it will be clear that the problem is not really about the number of tested hypotheses, but the ration between noise and signal. Next week I'll try to write about something more light hearted :)