Mathematics leaves very little room for fraud. Work can be incorrect, but the errors are generally on full display for the discerning reader to find. The most common fraud is stealing credit for another mathematician's work. It happens occasionally, but probably more often from sloppy literature searches than intent to deceive. The most interesting case of mathematical fraud that I have personally encountered arose in the humanities, in the course of a promotion review.
At most universities the promotion and tenure process has multiple stages. The candidate must first be approved by his department; after departmental approval, a broader university committee considers the candidate. The university Appointments and Promotions committee has less expertise than the departmental committee, but allows the university to monitor the departments' quality control, guarding against friendship overriding scholarly judgment and against mediocrity reproducing itself. Over a decade ago, I was asked by such a committee to help review a promotion decision at the university where I was then employed. The candidate was a tenured associate professor in the humanities being considered for promotion to full professor. To protect the guilty, let's name the candidate John Smith and pretend his field was intergalactic psychology. I was brought on to the case because the candidate's most important contribution, according to the outside letter writers, was his introduction of `new and important mathematical techniques' to intergalactic psychology, a field previously lacking in mathematical sophistication.
I read the candidate's book on mathematical intergalactic psychology. It introduced techniques from group theory (algebra) and catastrophe theory (topology) into intergalactic psychology.
For the nonmathematical reader, I note that group theory can be thought of as the study of symmetries of various sets. For example, consider a square with vertices in counter clockwise order: A,B,C,D. The simplest symmetry is rotation. For example, we can rotate the square, moving vertex A to vertex B's position, vertex B to vertex C's position, C to D's position, and D to A's position. Call that rotation, R. If I make the same rotation a second time, then I now have rotated vertex A to vertex C's position, B to D's position, C to A's position, and D to A's position. Symbolically, we write this new symmetry as RR, or better, R^2. The group of symmetries then consists of the set of all symmetries, with a multiplication defined on the elements of the set. The multiplication of two symmetries is defined simply by performing one symmetry after the other.
I immediately found a problem in the candidate's book. In his application of group theory to his theory of intergalactic psychological interactions, the candidate merely counted the number of members in the interaction. He defined no multiplication table on the interactions. Then he said that because the number of members was the same as the number of elements of some group, it must have the same multiplication defined as the group. Totally bogus. This half of the book had no intellectual merit whatsoever. It was gross academic negligence.
The second half of the book was devoted to the introduction of topological techniques to intergalactic psychology. Here I found a new problem. The first chapter of Part II was gibberish. Mathematical words were juxtaposed in a meaningless jumble. I couldn't make any sense of it. Perplexed, I looked up the references. Finally, I discovered what had occurred. It was the most bizarre form of plagiarism I had ever encountered. Prof. Smith had copied verbatim a popular article of a famous topologist, then recast it "in his own words" after the fashion of a grade school child writing a paper based on an encyclopedia entry. He had simply altered one or two words in each sentence, leaving most of the rest unchanged from the original. Unfortunately for Prof. Smith, mathematics is a very precise language. In his casual word substitution, he had rendered the entire article into gibberish. It made no sense. To make matters worse, he had not indicated that he was paraphrasing (?) published work. He merely listed the source in his bibliography.
So, I returned to the Appointment and Promotion Committee with the information that the candidate was a plagiarist and was guilty of either gross academic fraud or indefensible stupidity. The literature professors on the commttee were not bothered by the candidate writing gibberish. They thought it was simply creative to find new - albeit unknown - meanings for mathematical terms. Many of the committee members observed that the fraud had occurred in the work previously evaluated for tenure; our job was to evaluate the subsequent work, which the outside reviewers had praised. I argued that we should remove a fraud whenever discovered. Moreover, the outside reviewers had praised the fraudulent work above all the rest. Didn't this impeach the reviewer's judgment of the subsequent work? Many committee members felt we should not place our own judgment above that of the outside reviewers. Finally, I prevailed. The committee voted against promotion.
The University overruled us, and the professor was promoted.