OptoDrum - Testing Blind Animals

OptoDrum experiments consist of individual trials. During a trial, we test if the animal can see a certain stimulus condition (i.e., a certain difficulty, set by the resolution and contrast of the stripe pattern). If the animal passes such a test, i.e., if the optomotor reflex has been triggered during that stimulus condition, we then continue to a more difficult stimulus condition. Eventually, stimuli become too difficult, and this is how we find the visual threshold of the animal.

Blind animals should never pass any such trial, no matter how easy the stimulus is. However, like with non-blind animals, also for blind animals we expect that a certain fraction of trials will yield a false-positive result. That fraction is usually very low, maybe on the order of 5%. Still, it can happen, and that is why the OptoDrum software allows to require a certain number of confirmations.

We normally recommend a setting of

• 2 confirmations for ✓ successful trials
  This means: We only accept the conclusion "the animal can see condition x" if we get 2 successful trials at condition x. Obviously, the purpose of those "positive" confirmations is to capture potential false-positive trial results and guard against over-estimation of the animal's visual abilities.

At the same time, we also recommend a setting of

• 3 confirmations for ❌ failed trials
  This means: We only accept the conclusion "the animal cannot see condition x" if we get 3 failed trials at condition x. The purpose of those "negative" confirmations is to capture potential false-negative trial results and guard against under-estimation of the animal's visual abilities. It is actually relatively common to get false-negative trial outcomes (even if the animal can see the stimulus), in particular because we try and keep individual trials short to achieve a high-throughput experimental design. We therefore recommend to NOT reduce the required number of negative confirmations below 3.

In combination, those standard settings mean that we accept the conclusion "the animal can see condition x" if – at this condition x – we get 2 successful trials before we get 3 failed trials.

So, in a blind animal, how likely would we get 2 successful trials before we get 3 failed trials? Well, this depends on how likely it is to get a false-positive trial outcome in the first place. You can calculate this here:

Let's assume for now that we know the false-positive rate per trial for our blind animals. You can recreate the following examples, and any other combination, with the calculator above.

• Assumption: False-positive rate (per trial) of 5%.
  With the standard settings (2 positive and 3 negative confirmations) we would find for 1.4% of the animals that they can see, even though they are blind. This is likely an acceptable error rate.
• Assumption: False-positive rate (per trial) of 10%.
  With the standard settings (2 positive and 3 negative confirmations) we would find for 5.2% of our animals that they can see. If this is not an acceptable error rate in your project, you can increase the number of required positive confirmations to 3. This would reduce your overall error rate to 0.86%.
• Assumption: False-positive rate (per trial) of 20%.
  With the standard settings (2 positive and 3 negative confirmations) we would find for 18% of the animals that they can see. With 3 positive confirmations, the overall error rate would drop to 5.8%, and with 4 positive confirmations to 1.7%.

Such high false-positive rates per trial are of course less than ideal. Now, we will look at how we can determine the actual false positive rate per trial (Solution, Part II), and how we can reduce it (Solution, Part III).

It is quite easy to determine the false positive rate. In the OptoDrum, set the stimulus-contrast to 0, and run many trials, ideally with several individual animals of the type that are used in your study. We will call these trials the "baseline trials" in our further discussion. The zero-contrast stimulus is guaranteed to not be visible to your animals, and any successful trial is a false-positive trial. With many baseline trials, you get a good estimate of the false positive rate.

Below in Part III, we will discuss software settings to reduce the false positive rate if necessary. The adjusted settings can (and should) be based on your findings in the baseline trials. For this purpose, it is helpful to obtain a rich data set and to run the baseline trials with some unusual settings.

• Contrast: 0
  As mentioned above, the baseline trials should be run at 0 contrast.
• Threshold: 100
  Normally, the threshold is 1. (A trial is "successful" as soon as the trial-score as larger than the threshold). Consequently, any baseline trial that reaches a score > 1 is a false-positive trial. Increasing the threshold makes it harder to get a successful trial. Thus, increasing the threshold is one possible strategy to reduce the likelihood of false-positive outcomes. We want our baseline trials to teach us what a reasonable threshold would be. We therefore do not want our baseline trials to end as soon as the score hits 1, but we want to observe the further development of the score. We achieve this by making the threshold very large (100), to ensure that none of the baseline trials ends prematurely. Later, in our analysis, we can look at the scores of all baseline trials, and determine the false-positive rate for different values of the threshold (1, 1.1, 1.2, ....).
• Still-sitting time: 40 s
  "Still-sitting time" is the parameter in the OptoDrum software that determines the maximal duration of a trial (this is not the absolute duration of the trial, but the time within a trial deemed "relevant" by our algorithm.) If the score has not yet crossed threshold within that time limit, then the trial has a failed outcome. Standard value for still-sitting time is 20s. Similar to the change in the threshold parameter, see above, we can gain more insights by making the baseline trials last longer. Due to the increase of the threshold  to 100 (instead of 1), all trials are virtually ensured to have a failed outcome. Doubling the value of the parameter "still-sitting time" will double the duration and give us more data per baseline trial.
  Note: While the data per baseline trial becomes richer, the downside is that you can only collect half the number of baseline trials in the same amount of time, e.g. during 1 day. Try and understand the additional insights you can gain by longer trials, described below, and weigh this against the advantages of getting twice as many (but only normal-length) baseline trials. Based on your judgement, you may decide not to increase the "still-sitting time" parameter when collecting baseline trials.

The false-positive rate may be completely acceptable, and no changes to the standard settings may be needed. However, if you find that your (blind) animals have a rather high false-positive rate, then the following parameter changes in the software could be very helpful.

After we have collected baseline trials, we can estimate the false-positive rate by looking at the maximal value of the baseline-trial scores within the first 20 s of "relevant experimental time" (20s is the normal value of the parameter "still-sitting time"). For example, if 20% of the scores reached 1 or larger, we can assume a false-positive rate of 20% (for threshold 1). If, at the same time, only 5% of the scores reached a value of 1.4, then we can lower our false-positive rate to 5% by running our study with an increased threshold value of 1.4.

Note: It is easy to understand why increasing the threshold-parameter would reduce the chances of getting false-positive trials. Furthermore, it is straight-forward to estimate the new false-positive rate that results from a different value of the threshold-parameter, based on the baseline trials. However, there is a strong disadvantage to adjusting the threshold-parameter, and you should first see if strategy 2 (below) reduces your false-positive rate sufficiently. The disadvantage is that outcomes of studies with different thresholds become less comparable to each other. For example, if you determine the visual acuity threshold of (wildtype) mice with different values of the threshold-parameter, you will likely get different end results (such as 0.4 cyc/deg vs 0.3 cyc/deg, when you increase the threshold-parameter from 1 to, let's say, 1.4).

In the box above, we have discussed why the animals may produce a high false-positive rate: They show behavioral patterns that contain increased amounts of rotational head movement. These behaviors are not triggered by a visual stimulus (and certainly not during zero-contrast baseline trials), and it is completely random if such a behavioral event coincides with the rotation of the stimulus (we would refer to this as a "tracking event" in a seeing animal). If it indeed does coincide then the score will go up. However, it is just as likely that the behavioral event coincides with the opposite direction of the stimulus rotation ("anti-tracking event"). In that case the score will go down – this is the nature of how the OptoDrum calculates the score. In principle, we would expect from a blind animal that these random tracking and anti-tracking events are equally likely, and that score-increases and score-decreases would cancel out in the long run. It is like throwing a coin: In the long run, you get 50% head and 50% tail.

The expectation for (very) long trials would therefore be that the score – while it might initially swing up and down – would eventually approach 0. Compare this to the expectation for an animal that can see the stimulus: There, the overall trend is for the score to continuously increase, driven by true tracking events where the animal follows the rotating stripe pattern. On top of this rising trend, the score might also swing up and down, driven by additional random events.

In our blind animals, the problem are the intermittent peaks that can cross the threshold, yielding a false-positive trial. We can, however, take advantage of our expectation that those peaks should only be temporary, unlike the robustly increasing score trend of seeing animals.

The second strategy to reduce the rate of false-positive trial results is therefore to require threshold crossings to be robust. This is done by entering a certain time in the software parameter "Score needs to stay above threshold" (for example: 2 sec or 3 sec). With such a setting, if the score crosses the threshold, but drops below the threshold again before that required duration has passed, then this threshold-crossing event would not be acknowledged.

In fact, it is a good idea to have this a standard setting for all animals, not only for blind animals. The (slight) disadvantage is that it would make successful trials longer by that amount of time, but that is compensated by avoiding (some) false-positive trials.

How do the baseline trials inform us about a good value? How long should we require the threshold to stay crossed? Let's consider this: The quantity of (anti-)tracking events will influence the behavior of the score (how often and how wildly the score varies during a trial). For example, if such events happen relatively rarely, the score may increase, but it may then take some time to increase or decrease again. On the other hand, if such events are common, changes in the score also happen more frequently, and "peaks" in the score tend to be shorter. From our baseline data, we are interested in the following statistics: how long does it take the score to drop again to the same value, after an increase? This can be measured for each time point where the score goes up, for all baseline trials. Sometimes, it may only take a fraction of a second for the score to go back down to the same value, sometimes it may take tens of seconds, sometimes it might not happen at all during the remainder of the trial. Overall, this data will provide a distribution of "peak durations", and that distribution (in some way or another) is a representation of the "natural" behavior of these animals. [Above, in Part II, we suggested that the baseline trials may be recorded with an increased duration for the "still-sitting time" parameter, to make the trials longer. With longer trials, you would get more data and details on the distribution of "peak durations".] How does this distribution help us? During a trial, whenever the score increases, this could be the event that pushes the score above threshold. The distribution tells us the expected duration until the score drops again below threshold. From the cumulative distribution, you can directly read the likelihood that within a certain time window, the score drops back again (e.g., within 1 sec: 3%; within 2 sec: 10%; within 3 sec: 15%; within 5 sec: 40%). If those would be the actual numbers, the conclusion would be that you can drop the false-positive rate by 40% if you require the score to remain above threshold for 5 sec (for example from 10% false-positive rate down to 6%).

If we stay with this example: It is a judgement call if it would be worth it to make each positive trial a minimum of 5 sec longer, or if it would be more efficient to simply require 1 additional confirmation for positive trials. With the estimated false-positive rates (based on the baseline trials), you can use the calculator above to determine the number of confirmations that would be appropriate in your case based on your false-positive rate, which in turn depends on the value of the threshold and on the duration that you require the score to remain above threshold.

Similarly, it is a judgement call, when acquiring the baseline trials, if it is worth it to make those trials longer to learn more details about peak-duration distribution. You can simply use 2 sec or 3 sec as a standard value for the "Score needs to stay above threshold" parameter. Increasing the value beyond that would likely make the positive trials rather long and inefficient. Instead, you can determine the false-positive rate in the baseline trials with that standard value.

If you feel that you get an excessive amount of false-positive trials, especially if you study blind animals, you can reduce that significantly by following these steps:

1. Determine the actual false-positive rate with a sequence of baseline trials, as described above in Solutions, Part II.
2. Using the False-Positive Calculator as a guide, you can judge if it is necessary to reduce the false-positive rate, or if you can run your study either with the standard settings, or with an increased value for positive confirmations.
3. Solutions, Part III describes the two strategies how you can reduce the false-positive rate if necessary.
4. With a good estimate of the "final" false-positive rate for individual trials (with all parameter settings adjusted), you can again use the False-Positive Calculator above to see how this translates into an overall error rate of (wrongly) judging blind animals as "seeing". Adjust, if necessary, the required number of positive confirmations to reduce the error rate to an acceptable level.

Important: If you use different parameter settings, based on these procedures, you should use those parameter settings for your whole project. For example, if your animals have a progressive blinding disease, and you find that "old" (= blind) animals have a higher false-positive rate that warrants adjustment of software parameters, then those parameters should be used for the complete project, for young and old animals, to maintain comparability of data.