Anyone with a superficial understanding of scientific method and what constitutes good science, should be able to immediately recognize the inherent problems with this test.
First and foremost is the issue of what exactly does the test measure? The ATTS website claims to measure stability, shyness, aggressiveness, friendliness, protectiveness, self-preservation. In theory, the testers consider the following during the test:
The breed of the dog (hereditary purpose)
The training the dog has received
The dog’s age
The dog’s gender
Whether it has been spayed or neutered
The dog’s physical health (dog in season)
Whether it is a house dog or kennel dog
Yet the pit bull advocates present the stats in such a way as to imply that higher scores equal less aggression and lower scores equal more aggression. According to Herkstroeter, “Just because a certain percentage of dogs in a certain breed fail, this does not necessarily indicate aggression. Dogs fail for other reasons, such as strong avoidance. If you look at our statistics just from a perspective of aggression or non-aggression, they can be very misleading.” Herkstroeter states that 95% of the dogs that fail, do so because they lack confidence to approach the weirdly dressed stranger or walk on the strange surface. The remaining 5% fail because they take longer than 45 seconds to recover from the gunshot or the umbrella. Still pit bull advocates continue to distort the meaning of the test.
Second, as per ATTS website: "Comparing scores with other dogs is not a good idea" and the test "takes into consideration each breed's inherent tendencies". Cocker spaniels are evaluated against a cocker spaniel standard, not against german shepherds (or at least in theory, they are not supposed to), yet pit bull advocates would have you believe that all dogs are evaluated against one another.
The third troubling aspect of this test is not only the lack of a random sample but what appears to be pit bull advocates openly conspiring to groom test candidates and cherry pick only those individual dogs that are likely to pass. This is a conscious decision done for the sole purpose of inflating the scores to improve the pit bull's image.
The fourth major problem is in the inherent bias of the testers. Pit bull owners, breeders and advocates are in the position to pass or fail dogs that are under heavy criticism for what their critics perceive as their innate viciousness. As you will see, the testers have much discretion and a vested interest in the outcomes of the tests and they do not apply the rules fairly or consistently. There is no quality control to ensure that the testers are consistent in how they grade dogs' behavior.
Fifth, the test acknowledges that breed of dog (hereditary of purpose) is factored into the dog's performance and score, yet dogs are not tested in the presence of other dogs. This is especially critical with dogs that were bred to fight.
Sixth, the ATTS apparently does not require papers for purebred dogs. It seems that you can report any breed you like and do not need to provide any registry papers to prove it. One thing I find especially interesting is the flexibility around the issue of purity in pit bulls. When pit bulls attack, they are mixes but when they pass the ATTS, they are purebred, no questions asked. Just last week, Drayton Michaels made a point to say that most pit bulls were in fact not purebred pit bulls. This is another favored defense tactic when pit bull attacks hit the news media. In my experience reading all of the pit bull forums for ATTS information, discussion about the test is non-existent among the game-dog.com demographic. This group of pit bull owners is obsessed with bloodlines and pedigrees, and these dogmen do not even broach the subject of temperament testing.
Seventh, the handlers are familiar with the test, they not only know what to expect, they practice it with their dogs. In the real world, things don't work out that way and much of a dog's reaction can be based on the handler's reaction to the real world "stressful" events.
Eighth, The American Temperament Testing Society is not an impartial, scientific organization discovering "truth". They openly state their position on their website:
"Because of breed-specific dog legislation and negative publicity associated with many breeds of dogs, temperament testing has assumed an important role for today's dog fancier. The ATTS Temperament Test provides breeders a means for evaluating temperament and gives pet owners insight into their dog's behavior. It can have an impact on breeding programs and in educating owners about their dog's behavioral strengths and weaknesses as well as providing a positive influence on dog legislation."