In recent years, the U.S. Patent and Trademark Office (“Patent Office” or “Agency”) has come under increasing scrutiny over inconsistent patentability determinations. In fiscal year 2017 alone, more than 8,000 patent examiners made more than 600,000 patentability decisions.1 There is mounting empirical evidence that these 8,000 patent examiners have sharply divergent grant rates, implicating concerns that the decision to grant a patent is driven not only by the merits of the invention but also by the examiner to which the application is randomly assigned.2 The concern regarding inter-examiner disparities is so pressing that it led at least one scholar to quip, “there may be as many patent offices as patent examiners.”3
The harms associated with inter-examiner disparities in decision-making are undeniable. To begin, the fact of wildly divergent grant rates among examiners is highly suggestive that the Patent Office is regularly getting the decision to grant or deny a patent wrong. Much is at stake with the application of legal patentability standards. The patent system encourages valuable innovations by granting patents on inventions that are novel and that represent more than a trivial advancement over the current scientific understanding. However, should patents be issued covering technologies that fail to meet proper patentability thresholds, there may be an insufficient level of spurred innovation to justify the key costs of extending patent protection: higher prices and restricted access to the patented invention. As a result, if examiners are allowing invalid patents to issue, these patents may impose the costs of the patent system on society without producing the commensurate innovative benefits.4 Alternatively, if patent examiners are routinely denying patents on valid inventions, then innovation incentives may be dampened. To the extent that future inventors can observe these erroneous patent denials, they will discount the value of participating in the patent system to reflect concerns that they too may have their patent improvidently rejected.5 Beyond implicating examination quality concerns, inconsistent examiner decisions also offend theories of administrative justice while also raising questions of equity.6
One of the primary mechanisms by which agencies bring uniformity to low-ranking official’s determinations is by subjecting their decisions to higher-level agency review. The Patent Trial and Appeal Board (“PTAB” or “the Board”), which sits in panels of three administrative patent judges, reviews the determinations of patent examiners and reverses those in which they believe the examiner has erred. PTAB has been the subject of increasing scholarly attention and just this past term the Supreme Court upheld the constitutionality of the Board.7 However, the impact of PTAB on examiner decision-making has eluded analysis. While empirical examinations of the role agency adjudicatory boards play in bringing consistency to agency determinations has received scant scholarly attention, the few studies to date focusing on this issue conclude that agency adjudicatory boards largely fail to perform this consistency-enhancing function.8
The dearth of empirical scholarship addressing these issues stems at least in part from the difficulty of measuring the behavior of low-level administrative actors and uniformity in their practices. The Patent Office is helpful in this regard given the predictability offered by the relatively homogenous nature of examiners’ jobs. In essence, examiners are tasked with reviewing patent applications and determining whether a patent should be granted covering the underlying invention, a decision that can readily be codified and recorded. Moreover, it is exceedingly rare to be able to match data covering the behaviors of low-level administrative agents with information regarding administrative board reviews. Fortunately, we were able to collect data on both the application-level decisions that examiners make and the adjudicatory decisions of PTAB and to link those data sources by unique identifier codes assigned to each patent application and issued patent.
At the core of this empirical exercise is a database covering the behaviors of over 15,000 patent examiners making over four million patentability determinations between 2001 and 2017.9 At some point over their careers, 64% of these examiners experienced a reversal of a rejection that they had made by PTAB (and its predecessor, the Board of Patent Appeals and Interferences (“BPAI”)). Reviews of rejections were possible over our entire sample period, whereas PTAB only began reviewing patent grant determinations in late 2012. Out of those examiners actively reviewed in the latter period, nearly 10% of them had experienced a PTAB challenge over the patentability of a patent that they had decided to grant. This Essay uses these data to begin to fill the above-noted gap in the literature regarding administrative boards and consistency in agency determinations by examining the extent to which PTAB helps to bring greater uniformity to patent examiners decisions.
Our findings are promising. To begin, we find evidence that applications reviewed by restrictive examiners—i.e., inherently rejection-prone examiners—are more likely to ultimately have a rejection that is appealed and reversed than applications reviewed by non-restrictive/lenient examiners. This result is encouraging from a uniformity-inducing perspective to the extent that one would not believe that PTAB’s reversal function would lead to convergence in behavior if, for some reason, the Board were targeting its rejection-reversal activities on examiners who were already rejecting at very low rates. Similarly, we find evidence that applications reviewed by lenient examiners—i.e., inherently grant-prone examiners—are more likely to be associated with a patent issuance that is the subject of a PTAB challenge than applications reviewed by more restrictive examiners. To the extent that PTAB’s post-grant challenges are more targeted at applications reviewed by high-grant-rate examiners, this fact further establishes a foundation by which PTAB may induce convergence in examiner behavior.
Of course, just knowing that PTAB is targeting its review efforts in this manner does not tell us that greater uniformity in behavior will ultimately follow. The second part of our empirical analysis tests, in turn, whether the targeted examiners’ behavior will converge—e.g., whether low-grant-rate examiners whose rejections were challenged will increase their grant rates towards the mean. We indeed find evidence of subsequent convergence in behavior. For instance, we find that, after being reversed by PTAB for an erroneous rejection decision, the affected examiner’s grant rate increases.10 However, the magnitude of these corrections is notably stronger in the case of PTAB challenges over rejection decisions than allowance decisions.
Collectively, both of these sets of findings—targeting of PTAB challenges on the appropriate end of the granting distribution and the subsequent corrections in examiner practices—suggest that PTAB is playing a notable role in unifying patent examiner decision-making. However, given the separation in time and the separation from the examination process itself, PTAB’s post-grant evaluation function is arguably playing a weaker role in this regard.
Part II proceeds by outlining the harms associated with heterogeneity in Patent Office outcomes. It then turns to analyzing how PTAB can provide a consistency-enhancing function to patent examiner patentability decisions. Part III delineates examiner disparities and introduces the predictions associated with PTAB’s role in enhancing consistency in patent examiner decisions, which serve as the hypotheses that will guide our empirical analysis. Part IV describes the data set and methodology utilized. The results of our empirical analysis are also presented in this Part. Part V begins to explore implications of our results.
A. Harms Associated with Heterogeneity in Patent Office Outcomes
In the past 20 years, the Patent Office has tripled the number of patent applications it processes annually.11 As the number of patentability determinations the Agency makes in a given year grows, so too have concerns over uniformity in Patent Office outcomes. There is mounting empirical evidence that patent examiner grant rates wildly diverge.12 The Agency has come under increasing criticism that the decision to grant a patent application is driven not only by the merits of the invention, but also by happenstance as to which examiner the application is randomly assigned.13
Commentators have only recently begun to identify the drivers of heterogeneity in patent examiner decision-making. For instance, we now know that examiner grant rates tend to rise over the course of examiners’ careers.14 More specifically, grant rates sharply increase when patent examiners get promotions associated with less time to review applications.15 A separate study has demonstrated that the year an examiner was hired had a lasting impact on his or her granting proclivities.16 Examiners who started working when the Patent Office had a more permissive granting culture have higher grant rates throughout their careers than those who started working when the Agency’s granting culture was more restrictive.17 The training that newly hired patent examiners receive also plays a strong role in shaping their granting tendencies.18
Although commentators are still seeking to understand the drivers of examiner heterogeneity, the harms associated with inconsistent patent determinations are well understood. Disparities in examiner decision making suggest that at least some patent examiners are “missing the mark.” Patentability standards are designed to generally parallel the economic justifications for patents—that is, a patent should not be granted to an invention that is not novel because such non-novel patents have the potential to impose the costs of the patent system on society without producing the commensurate innovative benefits.19 As a result, the consequences of examiners routinely erring in reaching patentability determinations can be substantial.20
Aside from the concerns that inconsistent examinations invoke regarding the quality of the review process itself, inter-examiner disparity may also erode confidence in the Patent Office by creating the appearance of unfairness and arbitrariness.21 The dominant theories of administrative justice, such as Jerry Mashaw’s theory of “bureaucratic rationality,” hold that uniformity in agency outcomes is an important goal.22 The appearance of arbitrary decision-making could diminish the incentives for innovation, as would-be applicants might decide to pursue other endeavors. Inconsistent patentability decisions are also worrisome solely from an equity standpoint.
B. Patent Trial and Appeal Board’s Consistency-Enhancing Function
Given this swath of harms associated with heterogeneity in agency outcomes, it is unsurprising that agency heads often try to cabin the discretion of agency officials in an attempt to harmonize outcomes.23 The Director of the Patent Office has several mechanisms to limit examiner discretion, including subjecting examiner decisions to higher-level agency review.24 This Section explores how the Patent Office’s adjudicatory tribunal, PTAB, functions to bring uniformity to patentability determinations.25
PTAB reviews examiner decisions, overturning those in which it believes the examiner has erred.26 PTAB is comprised of statutory members and over 300 administrative patent judges (“APJs”), who sit in panels that consist of at least three members.27 PTAB has the potential to bring uniformity to examiner decisions through two primary capacities: lawmaking and error correction. The Board acts in a lawmaking function when it provides clarity to indeterminate areas of the law by reviewing an examiner’s patentability decisions. Existing legal doctrines do not always adapt to emerging fields of science. When inventors begin to file applications on these new technologies, the Board may fill this legal void. The Board’s decision as to when genes and gene fragments meet the utility standard is one such example.28 Similarly, the PTAB can help to fill gaps in the law that exist after the federal courts announce new precedent. However cognizant courts are of the need to issue opinions that clarify the law’s demands for similarly situated parties, courts rarely eliminate the discretion of other decision-makers in applying those opinions. Ex parte Mewherter, in which the PTAB considered whether machine-readable storage mediums constituted patentable subject matter, illustrates this gap-filling role of the Board.29 The publishing of PTAB opinions, as well as the designation of select opinions as precedential, enhances the ability of the Board to fill this lawmaking role.30 Overall, by eliminating indeterminacy in the law and reducing the discretion of examiners, the PTAB’s lawmaking function helps to increase uniformity in Patent Office outcomes.
PTAB can also increase uniformity in examiner decision-making by reversing examiners who erroneously applied existing legal doctrine. In this error-correction role, the Board is not necessarily fulfilling a lawmaking function but instead correcting the decisional process of an examiner who inaccurately applied existing law. The internal appeal process may improve error detection by relying upon interested parties to identify when the lower ranking decisions are wrong and hence appeal them for higher-level reconsideration.31 This decision-reversal role is the mechanism of primary interest in this Essay. Not only might this decision-reversal role create more uniformity by detecting errant decisions as they occur, but it may also shape behaviors in a more prospective sense. That is, the possibility of reversal by PTAB may encourage examiners to comply with more prevailing norms to the extent that examiners place disutility on such reversals. The power of this mechanism—which one might think of as a “deterrence” channel—may be especially strong in the case of those examiners who have already experienced a reversal event.
As one of us has previously explored, PTAB’s ability to bring homogeneity to examiner decisions was historically highly skewed.32 Many agency officials’ decisions can be appealed within the agency by two sets of constituents, enabling the agency’s adjudicatory board to correct the full spectrum of agency error. Take, for example, permit decisions initially made by one of the Environmental Protection Agency (“EPA”) Office of the Regional Administrators.33 If the EPA Regional Administrator decides to reject a permit to discharge pollutants, the aggrieved applicant can appeal the decision to the Environmental Appeals Board (“EAB”) and argue that the agency’s permit requirements are too restrictive or that the EPA Regional Administrator misapplied the existing requirements to the detriment of the applicant.34 On the other hand, if the EPA decides to grant the permits to discharge pollutants, then interested parties, such as environmental groups, can appeal the decision to the EAB and argue that the agency permitting criteria are too permissive or that the EPA Regional Administrator misapplied the existing requirement to the detriment of the interested party.35 In such a case, the possibility of symmetrical review gives the agency’s adjudicatory board the opportunity to correct the full spectrum of error.
In contrast, because the Patent Office’s adjudicatory board historically only had statutory authority to review examiner decisions to deny patents, its ability to bring uniformity to examiner decision-making has mostly been one-sided over time.36 If a patent examiner denies an application, disgruntled applicants can appeal the decision to the Board.37 In contrast, if the patent examiner granted the patent, there had historically been no robust mechanism before the Agency to challenge the grant of a patent.38 As a result, the Board could help harmonize examiner decision-making, but this function was limited to correcting those examiners that were overly restrictive—i.e., examiners rejecting too many valid patents. Because the Board did not have the authority to review patent grants, it could do little in the past (including the recent past) to correct overly permissive examiners—i.e., examiners granting too many bad patents.
That changed in 2011 (effective late 2012) when Congress created three new adjudicatory proceedings that provide third parties with a procedurally robust, streamlined way to contest the validity of a patent grant at PTAB.39 These new proceedings—known as Post-Grant Review, Inter Partes Review (“IPR”), and the transitional program for Covered Business Method Review—have been immensely popular, requiring the Patent Office to triple the number of administrative patent judges to handle the influx of petitions.40 As a result, both sides of patent examiners’ decisions—whether to grant or deny a patent—are now subject to higher-level agency review. In theory, this change should allow PTAB to rein in not only overly restrictive but also overly permissive examiners, giving the Board the ability to correct agency error across the full spectrum of decision-making.
While the Patent Office’s adjudicatory board has been the subject of increasing scholarly attention, little is known as to how well PTAB harmonizes patent examiner decision-making.41 Empirical investigations of PTAB to date have largely focused on the outcomes of the adjudicatory board and the Board’s interaction with federal court litigation—not the effect of the Board on examiner decision-making. For instance, Saurabh Vishnubhakat, Arti Rai, and Jay Kesan examined the extent to which these new proceedings provide a substitute for Article III patent validity litigation.42 Brian Love and Shawn Ambwani reported outcomes of IPRs and their “impact on co-pending patent litigation.”43 This empirical shortcoming is not unique to the patent literature. In fact, the extent to which higher-level agency review helps to unify agency outcomes has eluded analysis in the broader administrative law literature as well.44 Similar to the empirical investigations of PTAB, empirical studies of agency adjudicatory boards have tended to focus on the Board’s outcomes rather than its role in aligning low-ranking official determinations with agency policy.45
A. Patent Examiner Heterogeneity: Do PTAB Appeals Focus on Outliers?
The average grant rate of a patent examiner is roughly 67% over our full sample period. The granting tendencies of patent examiners—the rates at which they allow applications—vary dramatically around this mean. The standard deviation of examiner-specific grant rates across examiners is roughly 20%. Normalizing this variation by its mean level, this implies a coefficient of variation in examiner grant rates of roughly 0.30. Overall, this suggests a magnitude of examiner heterogeneity and allows us to easily reject the hypothesis that the degree of variation in grant rates we observe across examiners arises from chance alone. When focusing on the full sample of individual patent applications and regressing the incidence of the application being granted on a set of fixed effects for the various examiners, we estimate an F-statistic of 55 for the joint significance of the fixed effects. With this statistic, we can reject, with a greater than 99% level of confidence, the hypothesis that there are no differences in granting practices across examiners.
In Figure 1, we attempt to show the findings above graphically. We start in Figure 1A by setting forth a kernel density plot depicting the probability distribution of examiner grant rates across the full extent of the examiners included in our sample. As can be seen from Figure 1A alone, examiner grant rates vary widely across nearly the full spectrum of possible rates. In Figure 1B, we show what this distribution might look like by chance. For these purposes, we take each individual application across our sample and assign it a random number between 0 and 1. Based on this number, we derive a placebo indicator for whether or not the given application is granted—i.e., by setting the placebo indicator equal to “1” for values of the randomly assigned number that fall below the true mean grant rate across the sample (0.67). For each examiner, we then derive a mean rate of these placebo grants and thereafter set forth in Figure 1B a kernel density plot depicting the distribution of examiner placebo grant rates. By comparing these distributions, it is evident that the true rate of variation in grant rates across examiners varies considerably more than the variation than we might predict to occur if granting outcomes were randomly and independently determined across each individual application.
Note: This figure presents a kernel density plot (Epanechnikov kernel with “optional” bandwidth) of examiner grant rates across all examiners in the sample.
Note: This figure presents a kernel density plot (Epanechnikov kernel with “optional” bandwidth) of placebo grant rates across all examiners in the sample.
It is important to note that patent applications are randomly assigned to examiners within an Art Unit—a group of 15 to 20 patent examiners who review applications in the same area of technology. As a result, Figure 1A’s variation is not merely a result of differences in the patent worthiness of applications being assigned to patent examiners.46 One may nonetheless be concerned that Figure 1A merely reflects differences in underlying grant rates across technology groups, as opposed to differences across individual examiners. Accordingly, in Figure 2, we show the degree of variation in grant rates across examiners, but we instead capture the inherent, fundamental granting proclivity of examiners. For these purposes, we assign a grant rate to each examiner that effectively partials out the influence of other factors that are likely (or known) to affect examiner grant rates and that are themselves likely to vary across examiners. Such factors include the technology groups to which the examiners are assigned, the year in which the examiner disposes of the underlying application, the entity size of the applicant (e.g., an indicator for a large-entity applicant), the experience level of the examiner, and the pay grade of the examiner on the General Schedule (“GS”) pay scale (which affects the amount of time examiners have to review applications). To achieve this effect, we use the sample of individual applications and regress the incidence of the application being granted on a set of dichotomous variables reflecting each of these factors along with a set of examiner fixed effects. In Figure 2, we then depict a kernel plot of the distribution of estimated examiner fixed effects across the various examiners in our sample. As demonstrated, while this somewhat dampens the degree of spread in grant rates across examiners relative to Figure 1A, it continues to reflect a far greater degree of variation than one would predict by chance alone.
Note: This figure presents a kernel density plot (Epanechnikov kernel with “optional” bandwidth) of estimated examiner fixed effects across all examiners in the sample. Examiner fixed effects are derived from the predicted values from a regression of the incidence of the application being granted on a series of an examiner fixed effects, along with year effects, examiner GS levels, examiner experience levels and various application-level characteristics (large entity status of applicant, foreign priority status of applicant, and duration of examination and its square).
If PTAB were to be applied in a way that would counteract this cross-examiner heterogeneity and promote the uniform application of patent law, one would first expect that PTAB would target its evaluative activity on those examiners in greatest deviation from the norm. In other words, if PTAB hoped to facilitate convergence in examiner behavior, we would expect that its rejection-reversal efforts be focused on low-grant-rate / high-rejection-rate examiners and its grant-reversal efforts be focused on high-grant-rate examiners.
Hypothesis 1(A): The likelihood that an examiner ever experiences a PTAB (or BPAI) reversal of an application rejection will be higher for examiners at the lower ends of the examiner-grant rate distribution.
Hypothesis 1(B): The likelihood that an examiner ever experiences a validity challenge to PTAB of a patent that he or she had previously allowed will be higher for examiners at the upper ends of the examiner-grant rate distribution.
We should note that Hypotheses 1(A) and (B) are driven, in part, by a mechanical relationship. Restrictive examiners are more likely to reject patents than the average examiner, which will lead to a greater probability of a rejection appeal even if PTAB were to randomly review all rejections at the same likelihood. Similarly, lenient examiners are more likely to allow patents, which will lead to a greater probability of a post-grant opposition even if PTAB were to randomly review all issued patents with the same likelihood. In any event, the mechanical aspect to this targeting likelihood is irrelevant to the animating question of this Essay—does PTAB bring homogeneity to examiner decision-making? As long as PTAB is focusing relatively more of its rejection-reversal efforts on low grant-rate examiners (and vice-versa), it holds the potential to converge practices. It is nonetheless important to test Hypotheses 1(A) and (B) given the possibility, for instance, that PTAB’s rejection-reversal efforts disproportionately target examiners already exhibiting grant rates at the upper end of the scale, a state of the world in which PTAB would not be expected to produce convergent pressures.
Moreover, we note that PTAB’s uniformity-inducing powers may go beyond this mechanical function. Perhaps PTAB focuses its rejection-reversal efforts on rejections completed by those examiners with low allowance rates. To explore this possibility of an even stronger targeting role of PTAB, we also test the following subsidiary hypotheses:
Hypothesis 1(C): The likelihood that an individual application rejection will become the target of a PTAB (or, previously, a BPAI) appeal will be higher for examiners at the lower ends of the examiner-grant rate distribution.
Hypothesis 1(D): The likelihood that an issued patent will become the target of a PTAB validity challenge will be higher for examiners at the upper ends of the examiner-grant rate distribution.
By limiting its focus only to the set of applications receiving at least one final rejection, Hypothesis 1(C) rules out the mechanical effect discussed above. That is, if PTAB were to review all rejections with an equal probability, then, by assumption, one would not predict any difference in PTAB activity rates across this sub-sample of applications. However, if PTAB were to go a step beyond and affirmatively target its reversal efforts on those examiners with especially low grant rates, then one would indeed predict to find the relationship hypothesized in 1(C).
B. Does PTAB Reversal Have a Long-Term Effect on Patent Examiner Decision-Making?
If PTAB brings more homogeneity to patent examiner decision-making, it may do so not only by correcting specific errant decisions of individual examiners but also by altering patent examiner behavior in the long term. One might predict that all examiners—whether or not they have experienced a PTAB evaluation—would alter their behavior in the direction of PTAB’s expectations given the possibility of a PTAB reversal should they deviate from those expectations. Such an effect is analogous to what legal scholars refer to as “general deterrence” in the criminal or tort law context.
One might predict that this effect is even stronger for those examiners who have specifically had their decisions reversed or challenged by the adjudicatory board in the past, perhaps due to the visceral nature of this personal experience. Under this latter channel—which is analogous to what criminal or tort law scholars would term “specific deterrence”—the affected examiners would be expected to respond to previous board reversal decisions (or even mere challenges) by prospectively changing their future behavior in the manner expected by the Board. For instance, a restrictive patent examiner who has had a patent rejection reversed by the Board may alter her examining behavior to become more permissive moving forward. She may respond as such because she fears PTAB reversal in the future—under the assumption that she places some amount of disutility on such an outcome—and because by granting at higher rates she lowers the probability of being subject to that scrutiny again.47
To summarize, should examiners desire to avoid Board reversals, one might expect to observe the following pattern of behavior: (1) upon the Board’s reversal of an examiner’s rejection decision (or upon a mere appeal of a rejection decision to the Board), the affected examiner’s grant rate should increase thereafter and (2) upon the Board’s invalidation of a patent issued by an examiner (or upon the filing of a challenge to the Board over the validity of a patent issued by an examiner), the affected examiner’s grant rate should decrease thereafter. Collectively, these predictions motivate our second testable hypothesis. If the targeting of the Board’s activities predicted in Hypothesis 1 above were to hold true and if responses predicted in Hypothesis 2 (see below) were to likewise hold true, then the net result would be one in which PTAB operates to induce convergence in examiner behavior. For instance, if PTAB were to focus its rejection-reversal activity on low grant-rate examiners and if rejection-reversal events cause examiners to increase their grant rates, then we might expect that the below-average grant-rate examiners will tend to look more like the average examiner over time.
Hypothesis 2(A): Following PTAB’s reversal of an examiner’s rejection decision (or, alternatively, following the filing of an appeal of a rejection decision to PTAB), the affected examiner’s grant rate will increase thereafter.
Hypothesis 2(B): Following PTAB’s invalidation of a patent previously issued by an examiner (or, alternatively, following the filing of a challenge with PTAB over the validity of a patent previously issued by an examiner), the affected examiner’s grant rate will decrease thereafter.
To explore the role of PTAB in inducing greater uniformity in examiner behavior, we draw on two key sources of data. First and foremost, we collect data on individual patent applications from the Patent Office’s Patent Application Information Retrieval (“PAIR”) database, covering over 3.9 million utility patent applications that were filed on or after March 2001 and that reached a final disposition by May 2017. Importantly, for each application in the PAIR database, we have information on the name of the examiner primarily charged with reviewing the application, along with information about the outcome of the application and the various proceedings that occurred throughout the examination process. For instance, for each application, we can determine whether or not the applicant appealed a rejection decision to the BPAI (pre-2012) or to PTAB (post-2012), in addition to the outcome of that appeal. In each case, we know the precise dates of these various events—e.g., the date in which a notice of appeal was recorded for the application.
To complete these data, we merge information on the future PTAB outcomes of those applications that culminate in a patent issuance (information that is not otherwise included in the PAIR database). Data on PTAB filings—specifically IPR filings—were graciously provided to us by Arti Rai and Jacob Sherkow. At the outset, we emphasize several caveats with these PTAB data. First, while we collect data on applications that are disposed of by the Patent Office through May 2017, we only have information on PTAB challenges that were filed up to March 2016. Second, we only have consistent data on the date on which PTAB challenge was filed, not the date on which the Board decided to formally institute (or not institute) the challenge and not the date on which the Board decided to invalidate the patent in question (should PTAB have ruled in that manner). Moreover, we also do not have data on whether the post-grant challenge was ultimately successful. As such, overall, we have a more limited set of information bearing on PTAB’s activities in the post-grant context relative to PTAB’s role in reviewing rejections by examiners.
Finally, through a series of Freedom of Information Act requests, we have also collected a range of additional information about the examiner working at the Patent Office. This includes information regarding the GS pay level of the associated examiner at the time of application disposition and the year in which the examiner joined the Patent Office.
B. PTAB Targeting Analysis
With these data in place, we now turn to testing Hypothesis 1. To recap, with our first hypothesis, we are effectively exploring whether PTAB is targeting its evaluative efforts on the appropriate outlying examiners.48 At the outset, we note that this targeting function involves more than just PTAB. In the case of evaluations of examiner rejections, the appeals process is of course initiated by the aggrieved applicants. In the case of evaluations of examiner allowances, PTAB post-grant evaluations are of course initiated by third party petitioners. Nonetheless, for the purposes of brevity below, we may simply collapse this inquiry into one in which we refer to PTAB’s role in targeting its evaluation efforts on certain types of examiners.
We start by exploring whether we see relatively stronger rejection-reversal activity by PTAB on examiners with stronger tendencies to reject applications in the first place. As stated above, one can think of this as a pre-condition for PTAB to act as a device to induce greater uniformity in behavior. The empirical exercise is rather straightforward. First, for each examiner, we create a dichotomous variable that equals “1” if an examiner has ever had a rejection reversed by PTAB (or BPAI) and “0” otherwise. Next, we group examiners into quartiles based on their overall career grant rates—the number of applications that they have allowed divided by the number of applications that they have disposed of. However, instead of using the simple mean grant rate for each examiner, we assign each examiner a measure indicative of their inherent granting proclivity by risk-adjusting those grant rates by other factors that may differ across examiners but that are arguably orthogonal to this targeting exercise—e.g., the technology group to which the examiner is assigned. In this light, we ensure that we are not simply comparing the likelihood of experiencing a PTAB reversal across examiners in different technology groups with fundamentally different grant rates.49
Now, to explore where PTAB is targeting its rejection-reversal activities, we regress the indicator variable for whether or not an examiner has ever been reversed on a series of indicator variables capturing the inherent grant-rate quartile in which the relevant examiner falls. We omit the indicator variable for the first quartile from the regression to serve as the reference group. Put simply, this regression allows us to explore the association between the likelihood that an examiner experiences a reversal of a rejection that he or she had issued and where that examiner falls in the distribution of granting tendencies across examiners. If PTAB were to hold the potential to create uniformity in examiner behavior, one would hope, as a first step, that its rejection reversal efforts be focused on those examiners who are more rejection prone.
We present the results of this exercise in Column 1 of Table 1. The reported coefficients suggest that the likelihood an examiner is reversed at some point over his or her career declines as we move up the various examiner grant-rate quartiles. For instance, the -0.29 coefficient estimated for the fourth quartile indicator suggests that the likelihood an examiner in the bottom grant-rate quartile has a rejection reversed at some point over her career is roughly 29 percentage points—or roughly 41% relative to the mean—higher than the likelihood that an examiner in the top grant rate quartile has a rejection reversed over her career. Similarly, the lowest-grant rate examiners are roughly 9.4 percentage points and 1.5 percentage-points more likely to have a rejection reversed at some point relative to examiners in the third and second quartiles, respectively. All told, these results are consistent with a story in which PTAB focuses its rejection-reversal efforts on applications reviewed by examiners who issue the most rejections.
Notes: * significant at 10%; ** significant at 5%; *** significant at 1%. Standard errors are reported in parentheses and are clustered to correct for autocorrelation within given examiners over time.
As discussed above, an outcome of this nature may arise even if, hypothetically, PTAB were to issue reversals at an equal rate across all applications receiving a final rejection. Nonetheless, one may wonder whether PTAB’s role in targeting low-grant rate examiners for rejection reversals goes beyond this hypothesized neutral stance and whether PTAB places a greater emphasis on low-grant rate examiners in its rejection-reversal decisions. To explore this, we simply look at the sub-sample of applications that received a final rejection and regress an indicator variable for whether the final rejection was reversed by PTAB on the series of indicator variables capturing the examiner-grant-rate quartiles, again leaving out the first-quartile variable to serve as the frame of reference. In the neutral-PTAB hypothesis, one would not predict that the likelihood that this final rejection would be reversed would differ depending on the granting proclivity of the assigned examiner. However, as presented in Column 2 of Table 1, we find that this likelihood falls as we move from the bottom to the top of the grant-rate distribution. Accordingly, our evidence suggests that the lowest-grant-rate examiners indeed attract greater PTAB rejection-based scrutiny, even above and beyond the fact that they are creating more opportunities for this scrutiny in the first place through their higher rejection rates.
In Table 2, we extend this analysis, but now look at matters from the opposite side. Do high grant-rate examiners attract post-grant challenges by PTAB at a higher rate relative to low-grant-rate examiners? The structure of this table and methodology underlying it largely parallels that of Table 1. One exception is that instead of recording whether examiners have experienced any reversals of their grant decisions over the course of their careers, we record whether examiners have experienced any PTAB challenges of patents they have issued. Overall, these PTAB post-grant challenge findings are consistent with PTAB rejection-reversal findings. If PTAB were to induce uniformity in examiner behavior, then one would predict that its post-grant challenge activities—i.e., challenges that would possibly lead to invalidating previously issued patents—would be focused on examiners at the upper end of the grant-rate distribution. We find that this precondition is indeed met. In particular, examiners in the fourth quartile of the inherent grant-rate distribution are roughly 19 percentage points more likely to have issued a patent that is the subject of a post-grant challenge at some point over their career relative to examiners in the first quartile.
As above, this relationship may in part be a mechanical by-product of the fact that the higher grant-rate examiners create more possibilities for post-grant challenges in the first place. To further test whether PTAB’s post-grant-challenges target high-grant-rate examiners, we estimate a specification analogous to that estimated in Column 2 of Table 1. In this alternative approach, we look for evidence of targeting of high-grant rate examiners while equalizing the opportunity for challenges. To do so, we focus on a sample of issued patents and regress an indicator variable for whether the relevant issued patent is ultimately the subject of a PTAB post-grant challenge on the various quartiles of the examiner grant-rate distribution. We continue to estimate an increase in the rate of the likelihood of a PTAB challenge as we move from the bottom to the top of the examiner grant-rate distribution, suggesting that PTAB is affirmatively focusing its post-grant challenge activity on the highest grant-rate examiners.
Notes: * significant at 10%; ** significant at 5%; *** significant at 1%. Standard errors are reported in parentheses and are clustered to correct for autocorrelation within given examiners over time.
C. Examiner-Response Analysis
Above, we found that low grant-rate examiners are more likely to experience reversals of their rejections. If those low-grant examiners were, in turn, to increase their grant rates after these reversals, then PTAB may indeed serve as a device that will lead to greater standardization of examiner practices over time. Similarly, we had found that high grant-rate examiners are more likely to experience post-grant challenges. If those high-grant rate examiners subsequently were to decrease their granting tendencies, then PTAB may also help to induce convergence from above. In this Section, we attempt to round out this empirical exercise by exploring how examiners respond prospectively to experiencing PTAB evaluations of their rejection and allowance decisions—e.g., do grant rates go up following an experience with a PTAB rejection-reversal process?
We approach this exercise by employing an event-study methodology whereby we track the grant rates of examiners over time, where time is measured with reference to the period leading up to an examiner experiencing a PTAB rejection reversal (or post-grant challenge) and the period of time subsequent to such an event. In particular, we show these event-time trends graphically where the time relative to the event—e.g., to the rejection reversal—is indicated along the x-axis and the examiner grant rate is indicated along the y-axis.
We acknowledge that an event-study exercise of this nature confronts various obstacles. The first such obstacle is that examiners may experience multiple events of interest over their careers—e.g., multiple rejection reversals by PTAB. To the extent there is possibly more than one such event per examiner, it is difficult to trace the average behavior across all examiners in the time leading up to and following rejection reversals or PTAB post-grant challenges in the graphical manner discussed above. If everyone were to experience a second (or third, etc.) event and if everyone experienced that subsequent event at the same length of time following the first event, then we could indeed graphically depict day-by-day how examiner behavior evolves on average as we approach the first event, the second event, etc. However, the reality is that not all examiners will experience a second event and those that do experience subsequent events do so at different lengths of time following the first, making it difficult in a single graph to trace average examiner behavior before and after the first event, before and after the second, etc. Given these challenges, we instead elect to trace examiner behavior before and after the first relevant event that they experience over their career—e.g., the first time they have a rejection reversed by PTAB.
The essence of this event study approach is to look for changes in behavior over “event time” in order to infer changes in behavior that result from the events themselves. This structure, however, poses a second key empirical challenge—i.e., the need to separate the effects of the event occurring from other factors that also change over time and may affect examiner grant rates. For instance, changes in the law respecting patentable subject matter over time may lead to changes over “calendar time” in examiner grant rates. Relatedly, as we think about the determinants of examiner behavior, another factor that naturally changes over time is their level of experience, which may also alter examiner behavior considering the possibility that examiners will learn over time how to more efficiently conduct examination reviews.50 Examiners also receive promotions over time, which our prior research has shown strongly impacts examiner grant rates due to the changes in examination time allocations.51
To address these concerns, we use multi-variate regression techniques to estimate the relationship between examiner grant rates and the relevant event time—e.g., the proximity of the time in which the application is disposed of relative to the time of the examiner’s first rejection reversal by PTAB—while controlling for the above-mentioned factors. In more specific terms, using our application-level data, we regress the incidence of the application in question being allowed by the examiner on a set of indicator variables capturing different windows of time around the relevant PTAB event—e.g., indicator variables for whether the application in question was disposed of by the relevant examiner in the period of time between 0 and 200 days following the examiner’s first PTAB rejection reversal, between 200 and 400 days following the examiner’s first PTAB rejection reversal, etc. Indications for periods of time leading up to the event are also included. To control for the other factors that may likewise change over time and potentially confound this analysis, we include the following controls: (1) application-disposition-year-by-month fixed effects (i.e., indicator variables for each separate calendar month-year combination over the sample, assigned based on the timing of the disposition of the application), (2) examiner-experience fixed effects (in years), and (3) examiner GS pay-level fixed effects.52
Since examiners start at the Patent Office in different years, get promoted at different stages of experience, and experience their first PTAB evaluation event at different periods of time, it becomes statistically possible to separate out the effects of event time from these other factors. Helpful in this regard is the fact that we would be drawing on nearly four million patent applications over a roughly 16-year period to achieve this statistical separation. In the figures that follow, we will present the results from these regressions graphically. To do so, we plot the estimated coefficients of the indicator variables for the different event-time bins. In the process, we deliver on the graphs promised above—e.g., plotting the time leading up to and following PTAB evaluation event of interest on the x-axis and the examiner grant rate on the y-axis—while adjusting the depicted grant-rate trend for other related factors that likewise change over time, allowing us to better interpret any changes in this trend around the time of the event as arising from the event itself. We further note that the regressions producing these graphs drop the indicator variable representing the 200 days leading up to PTAB event such that this time period will serve as the reference period in the presented graphs.
In the first event-study graph that we present, Figure 3, we focus on exploring how examiner behavior changes after examiners experience their first board reversal at PTAB (or BPAI). The timing of the event itself in Figure 3 is based on the date of PTAB’s reversal decision. As depicted, we find that grant rates indeed jump upwards following this first reversal event by close to two percentage points—or by upwards of three percent relative to the mean grant rate. Coupled with the above findings that PTAB targets its rejection reversal efforts at below-grant-rate examiners, these findings of increased grant rates following an examiner’s first PTAB rejection reversal imply that PTAB may indeed help induce convergence in examiner grant rates from the bottom of the distribution.
Figure 3, however, does present one important concern. It appears grant rates may have been trending upward prior to the timing of the first rejection reversal. This fact raises a concern that the increased grant rates were caused by some factor omitted from the model, as opposed to the reversal event. On the other hand, this pre-trend may still be consistent with a causal effect of PTAB reversals and may simply reflect PTAB (or BPAI) proceedings that occurred in the time leading up to the final decision. The time between filing rejection appeals and handing down final decisions is considerable. During the interim, while the Board is examining the decision and calling upon the examiner to provide certain answers regarding the basis for their decision, the examiner may receive various signals from the Board that likewise cause them to adjust their behavior—e.g., to be more permissive in the case of overly restrictive examiners.
Figure 3. Event-Study Analysis: Relationship between Examiner Grant Rates and Time Leading up to and Subsequent to Date of Examiner’s First Rejection Reversal at PTAB (or BPAI)
In Figure 4, we explore whether examiners are indeed altering their behavior based on occurrences that transpire earlier in the appeal process. For these purposes, we now treat as the relevant “event” the first time that an examiner experiences an appeal to the Board of a rejection decision, where the timing of the event is based on the timing of the notice of appeal. In this alternative approach, we ask how an examiner’s behavior changes around the time of their first ever experience with PTAB (or BPAI) rejection-evaluation process. As demonstrated by Figure 4, we no longer find that grant rates were trending upwards prior to the examiner’s first rejection-appeal event, but instead that the increase in their granting tendencies appears to occur at the moment of the rejection-challenge event itself. Subsequent to that event, the examiner’s grant rate appears to rise monotonically over time.
The above figures are similar in that they model behavior before and after a single event. As discussed above, some examiners experience multiple encounters with PTAB over their careers. While that fact complicated the ability to structure an event-time figure in terms of days leading up to and following multiple events of this nature, it is nonetheless possible to structure an event-time figure of an alternative nature. In this alternative graphical approach, we simply plot an examiner’s grant rate on the y-axis against the sequence of rejection reversals on the x-axis. The event-time of significance in this approach is thus the number of rejection reversals as opposed to days before and after a reversal. We present the results of this exercise in Figure 5, where the time before the first rejection reversal is the reference period. We follow examiners through four reversals. To ensure balance in the process, we focus on examiners that experience at least four rejection reversals over their career. Admittedly, this is a select group, as only 19% of examiners have at least this many reversals over their career. We generated the results depicted in Figure 5 from a regression that allows us to estimate the relationship between grant rates and rejection-reversal sequencing while controlling for the same set of time-varying controls included in the above regressions. As demonstrated, we find that grant rates jump by over two percentage points after the first reversal, consistent with the above findings. Thereafter, grant rates continue to rise monotonically with subsequent reversal events.
All told, the results paint a strong and consistent picture that grant rates rise subsequent to experiences with a PTAB reversal of a rejection decision. This finding reinforces the conclusion that PTAB may alter examiner behavior in a way that causes examiners to exhibit less heterogeneous behavior.
Figure 4. Event-Study Analysis: Relationship between Examiner Grant Rates and Time Leading up to and Subsequent to Date of Examiner’s first Filing of Notice of Appeal of a Rejection Decision to PTAB (or BPAI)
Figure 5. Relationship between Examiner Grant Rates and Sequence of Rejection Reversal Among Examiners that Experience at Least 4 Reversals over their Career
In our final empirical exercise, we now turn to evaluating how examiners respond to their first encounter with PTAB post-grant challenge. For these purposes, we employ an event-study methodology similar to that explored in Figures 3 and 4. As discussed above, we are limited in our ability to perform as rich of an analysis in this setting for a number of reasons, including the fact that PTAB only began to allow such challenges in late 2012 and by the fact that the data we have collected only have the date of PTAB challenge filing itself.53 With these caveats in mind, we present the results of this exercise in Figure 6, exploring how grant rates evolve in the periods leading up and subsequent to the filing date of an examiner’s first challenge against a patent that he or she issued. As demonstrated by this figure and as predicted by Hypothesis 2, grant rates do fall subsequent to this first PTAB post-grant event. However, the magnitude is far smaller than PTAB rejection-reversal results. By two years following the event, grant rates have only fallen by about 0.5 percentage points and even at that date we cannot reject that there was no change at all given the width of the estimated confidence intervals. By the time we have reached 1200 days (just over three years) following the first post-grant challenge, grant rates have fallen by roughly 1.5 percentage points (now statistically distinguishable from zero). Accordingly, there is some evidence to suggest that examiner grant rates do fall after PTAB has been called upon to evaluate a patent grant that it had issued in the past. However, this post-grant-evaluation reaction appears much less sensitive than PTAB rejection-evaluation response.
Figure 6. Event-Study Analysis: Relationship between Examiner Grant Rates and Time Leading up to and Subsequent to Date of Examiner’s first Post-Grant Challenge at PTAB
V. Normative Considerations and Conclusion
Our findings suggest that PTAB is bringing more consistency to patent examiner decision-making. More specifically, our results suggest that patent grants by lenient examiners are more likely to be appealed and reversed by PTAB than patent grants by non-lenient examiners. Our results also demonstrate that patent denials by restrictive examiners are more likely to be appealed and reversed by PTAB than patent denials by non-restrictive examiners. Finally, our findings also suggest that rejection reversals by PTAB results in granting tendencies to rise and patent invalidations by PTAB results in granting tendencies to fall. Collectively, these results imply that PTAB causes restrictive and lenient examiners to move toward the average grant rate of patent examiners. In other words, PTAB has a unifying effect on patent examiner’s decision-making.
This is all good news, but even with PTAB’s consistency-enhancing effect, there has been increasing concern that patentability decisions at the Patent Office are inconsistent. This Part thus considers the ways in which PTAB could further enhance the ability of the adjudicatory board to bring uniformity to examiner decision making.
While our findings suggest that PTAB is bringing uniformity to patent examiner decisions, our results also suggest that the adjudicatory board is better able to induce convergence in the behavior of overly restrictive examiners rather than overly permissive ones. We attribute this trend to several factors. First, the adjudicatory board does not have as many opportunities to correct overly permissive examiners as it does overly restrictive examiners, as patent denials are more frequently challenged in a PTAB proceeding than patent grants. In fiscal year 2017, PTAB decided over 14,000 appeals from patent denials and approximately 1,500 challenges of patent grants.54 The lopsided nature of challenges before PTAB stems from several factors, including the fact that unlike patent denials, the validity of issued patents can also be directly challenged in federal district court. Although PTAB has proved to be a more popular forum for challenging granted patents than initial projections suggest,55 the majority of issued patents are still challenged only in federal court.56 Moreover, because post-grant PTAB proceedings are more formal and trial-like than patent-denial PTAB proceedings, the former are much more expensive than the latter.57 The difference in costs also likely contributes to the skewed review rates.
It is difficult to utilize PTAB to more effectively target appeals of permissive patent examiners because the adjudicatory board relies on third parties to challenge the validity of a patent at the Board. As a result, the Patent Office should consider other methods to review the patentability decisions of permissive patent examiners. Currently, the Patent Office conducts a random sampling method of patent examiner decisions that are subject to further quality review.58 We recommend that the Patent Office sample more heavily from patent examiners whose grant rates are in the top quartile of the Agency in an effort to provide further consistency to examiner determinations.
The second reason why PTAB’s ability to bring uniformity to examiner decision-making is skewed stems from our results suggesting the specific deterrence effect of reversal is stronger for patent denials than patent grants. More specifically, our findings suggest that examiners who have a patent denial reversed show a greater response in their grant rates than examiners who have a patent grant reversed.59 There are a number of reasons a patent examiner may be more responsive to the reversal of a patent denial than a patent grant. Perhaps most saliently, patent examiners are directly involved in PTAB proceedings involving patent denials. Examiners whose patent rejection is appealed to the adjudicatory board meet with several senior examiners to review the rejection decision and write and file a brief outlining their reasons why they rejected the application with PTAB.60 If the adjudicatory board reverses its decision, the application is returned to the examiner to issue the patent grant. In contrast, patent examiners are not part of the post-grant PTAB proceedings. That is, they do not file any materials with the adjudicatory board defending their decision. In fact, PTAB determination is never forwarded directly to the examiner in question.
To increase the ability of PTAB to bring uniformity to overly permissive patent examiners, we encourage the Patent Office to increase the flow of information from post-grant proceedings to patent examiners. The Agency has made moves to bridge the gap between post-grant PTAB proceedings and patent examiners with a 2011 pilot program designed to enhance communication between patent examiners and PTAB.61 The pilot program, which included feedback on best practices, focused on providing patent examiners, who were reviewing patent applications related to an ongoing PTAB trial proceeding, information on prior art and arguments.62 We applaud the Agency for taking this critical first step and encourage the Patent Office to continue to increase the flow of information between PTAB and patent examiners. By providing patent examiners with direct feedback on cases involving patents they issued as well as providing best practices or common mistakes to all patent examiners, the Patent Office can harness the adjudicatory board’s ability to bring more uniformity and improve patent examiner decision-making.
. U.S. Patent & Trademark Office, Performance & Accountability Report: FY 17, at 10, 168 tbl.1 (2017) [hereinafter USPTO PAR 2017], https://www.uspto.gov/sites/default/files/
. Sean Tu, Luck/Unluck of the Draw: An Empirical Study of Examiner Allowance Rates, Stan. Tech. L. Rev., Oct. 2011, at 1, 6–7; Michael D. Frakes & Melissa F. Wasserman, Knowledge Spillovers and Learning in the Workplace: Evidence from the U.S. Patent Office 15 fig.1 (Duke Law Sch. Pub. Law & Legal Theory Series, Working Paper No. 2018-11), https://papers.ssrn.com/sol3/papers.
. Iain M. Cockburn et al., Are All Patent Examiners Equal? The Impact of Characteristics on Patent Statistics and Litigation Outcomes, in Patents in the Knowledge Based Economy 28 (Wesley M. Cohen & Stephen A. Merrill eds., 2003).
. The harms associated with invalid patents are substantial. For a summary, see Michael D. Frakes & Melissa F. Wasserman, Does the U.S. Patent and Trademark Office Grant Too Many Bad Patents?: Evidence from a Quasi-Experiment, 67 Stan. L. Rev. 613, 618–21 (2015) (summarizing the harms associated with invalid patents).
. Stephen Yelderman, The Value of Accuracy in the Patent System, 84 U. Chi. L. Rev. 1217, 1224–25 (2017).
. Jerry L. Mashaw, Bureaucratic Justice: Managing Social Security Disability Claims 25–26 (1983); see also Robert A. Kagan, Inside Administrative Law, 84 Colum. L. Rev. 816, 820 (1984) (reviewing Jerry L. Mashaw, Bureaucratic Justice: Managing Social Security Disability Claims (1983)) (detailing how Mashaw’s “bureaucratic rationality” is a model of agency adjudication that facilitates “[g]reater control and consistency” by placing “the overriding value” on “accurate, efficient and consistent implementation of centrally-formulated policies”).
. Oil States Energy Servs., LLC v. Greene’s Energy Grp., LLC, 138 S. Ct. 1365, 1377
. David Hausman, The Failure of Immigration Appeals, 164 U. Pa. L. Rev. 1177, 1196 (2016) (concluding that the Board of Immigration Appeals is largely ineffective in bringing uniformity across immigration judges).
. See infra Section IV.A for more information about the data this Essay analyzes.
. See infra Section IV.C.
. Compare USPTO PAR 2017, supra note 1, at 10, 168, with U.S. Patent & Trademark Office, Annual Report 1998, at 13 (1998), https://www.uspto.gov/about-us/performance-and-planning/annual-reports/annual-report-1998-0.
. Tu, supra note 2, at 10–11; see also Frakes & Wasserman, supra note 2, at 15 fig.1.
. See Cockburn et al., supra note 3, at 24–25 (finding that differences in examiners explain a significant percentage of the variation in the characteristics of issued patents, and that some examiners are more likely than others to have their patents upheld in court); see also Douglas Lichtman, Rethinking Prosecution History Estoppel, 71 U. Chi. L. Rev. 151, 154–55 (2004) (finding that certain examiners more systematically required applicants to narrow the scope of their patents).
. Mark A. Lemley & Bhaven Sampat, Examiner Characteristics and Patent Office Outcomes, 94 Rev. Econ. & Stat. 817, 817 (2012); see also Michael D. Frakes & Melissa F. Wasserman, Is the Time Allocated to Review Patent Applications Inducing Examiners to Grant Invalid Patents? Evidence from Microlevel Application Data, 99 Rev. Econ. & Stat. 550, 550–51 (2017).
. See Lemley & Sampat, supra note 14, at 817; see also Frakes & Wasserman, supra note 14, at 550.
. Michael D. Frakes & Melissa F. Wasserman, Patent Office Cohorts, 65 Duke L.J. 1601, 1639 (2016).
. See, e.g., Frakes & Wasserman, supra note 4, at 618–21 (summarizing the harms associated with invalid patents).
. The harms associated with invalid patents are substantial. See id. at 618–21. The harms associated with the denial of valid patents are also substantial. For a summary, see Yelderman, supra note 5, at 1224–25.
. Michael Abramowicz & John F. Duffy, Ending the Patenting Monopoly, 157 U. Pa. L. Rev. 1541, 1558 (2009) (noting that the Patent Office’s “challenge is to ensure that the judgments of [its patent examiners] are of relatively high quality and highly consistent”); see Mashaw, supra note 6, at 73.
. Mashaw, supra note 6, at 25–26; Kagan, supra note 6, at 820.
. Christopher J. Walker & Melissa F. Wasserman, The New World of Agency Adjudication, 107 Calif. L. Rev. 141, 176–77 (2019).
. 35 U.S.C. § 6(a) (2012).
. Id. § 6(b). Administrative patent judges are “persons of competent legal knowledge and scientific ability who are appointed by the Secretary [of Commerce], in consultation with the [Patent Office] Director.” Id. § 6(a).
. Ex parte Fisher, No. 2002-2046, 2004 WL 2185929, at *3–14 (B.P.A.I. Apr. 1, 2004).
. Ex parte Mewherter, No. 2012-007692, at 4–7 (P.T.A.B. May 8, 2013).
. For a summary of how PTAB designated opinions as precedential, see Walker & Wasserman, supra note 23, at 191–96.
. See Steven Shavell, The Appeals Process as a Means of Error Correction, 24 J. Legal Stud. 379, 381 (1995).
. Technically, the previous work explored the federal courts review of the Board’s decision, but the same principle is at work in high lever agency review of agency official’s decisions. See, e.g., Melissa F. Wasserman, The PTO’s Asymmetric Incentives: Pressure to Expand Substantive Patent Law, 72 Ohio St. L.J. 379, 401–07 (2011) [hereinafter Wasserman, The PTO’s Asymmetric Incentives] (examining how the asymmetric review of the Patent Office’s prior adjudicatory board, Board of Patent Appeals and Interferences, created an incentive for the agency to adopt pro-patent interpretations of the Patent Act); see also Melissa F. Wasserman, Deference Asymmetries: Distortions in the Evolution of Regulatory Law, 93 Tex. L. Rev. 625, 655–58 (2015) (discussing how asymmetric review of agency decisions by federal courts can create a pro-regulatory constituency bias in the development of substantive law).
. 40 C.F.R. § 124.19(a) (2018).
. See id. § 124.19(a)–(b) (describing the process, including what parties, can file an appeal of a petition determination); see also Frequently Asked Questions, U.S. Envtl. Prot. Agency: Envtl. Appeals Bd., https://yosemite.epa.gov/oa/EAB_Web_Docket.nsf/General+Information/
Frequently+Asked+Questions?OpenDocument (last visited Apr. 27, 2019).
. 40 C.F.R. § 124.19(a).
. Wasserman, The PTO’s Asymmetric Incentives, supra note 32, at 401–06.
. 35 U.S.C. § 134 (2012) (outlining when a party may seek a Board appeal); see also Wasserman, The PTO’s Asymmetric Incentives, supra note 32, at 404–05 (discussing how the Patent Office’s adjudicatory board historically had the authority to review only patent denials).
. Wasserman, The PTO’s Asymmetric Incentives, supra note 32, at 404.
. Leahy–Smith America Invents Act, Pub. L. No. 112–29, 125 Stat. 284 (2011); see id. §6 (codified at 35 U.S.C. §§311–319, 321–329 (2012)) (describing “inter partes review” and other “post-grant review proceedings”); id. §18 (codified at 35 U.S.C. § 321) (describing the “transitional program for covered business-method patents”); see also Kent Barnett et al., Admin. Conference of the U.S., Non-ALJ Adjudicators in Federal Agencies: Status, Selection, Oversight, and Removal 15, 18 (Draft Report 2018), https://www.acus.gov/
sites/default/files/documents/Non-ALJ%20Draft%20Report_2.pdf (reporting there were 275 administrative patent judges as of 2017); Gene Quinn, Chief Judge Rader Swears in New Administrative Patent Judges, IP Watchdog (Jan. 25, 2012), http://www.ipwatchdog.com/
2012/01/25/chief-judge-rader-swears-in-new-administrative-patent-judges (noting there were approximately 100 APJs in 2011).
. U.S. Patent & Trademark Office, Performance & Accountability Report: Fiscal Year 2013, at 23 (2013) [hereinafter USPTO PAR 2013] (noting that the “tremendous inflow of new proceedings is higher than initially estimated”); see also Barnett et al., supra note 39, at 18, 22; Quinn, supra note 39.
. See generally, e.g., Rochelle Cooper Dreyfuss, Giving the Federal Circuit a Run for Its Money: Challenging Patents in the PTAB, 91 Notre Dame L. Rev. 235 (2015) (reviewing statistics and final decisions of the PTO); see also Timothy R. Holbrook, The Patent Trial and Appeal Board’s Evolving Impact on Claim Construction, 24 Tex. Intell. Prop. L.J. 301, 320–23 (2016) (evaluating potential impacts of PTAB on claim construction doctrine); Greg Reilly, The Constitutionality of Administrative Patent Cancellation, 23 B.U. J. Sci. & Tech. L. 377, 377 (2017) (assessing “claims that adjudication of the validity of issued patents in the [USPTO] is unconstitutional”); Walker & Wasserman, supra note 23, at 162–74 (situating PTAB adjudication in the modern landscape of agency adjudication).
. See generally Saurabh Vishnubhakat et al., Strategic Decision Making in Dual PTAB and District Court Proceedings, 31 Berkeley Tech. L.J. 45 (2016) (examining, among other things, the extent to which these new proceedings are being utilized as a substitute for Article III patent validity litigation).
. Brian J. Love & Shawn Ambwani, Inter Partes Review: An Early Look at the Numbers, 81 U. Chi. L. Rev. Dialogue 93, 95 (2014).
. One notable exception is Hausman, supra note 8, at 1181–86.
. See Gerald K. Ray & Jeffrey S. Lubbers, A Government Success Story: How Data Analysis by the Social Security Appeals Council (with a Push from the Administrative Conference of the United States) is Transforming Social Security Disability Adjudication, 83 Geo. Wash. L. Rev. 1575, 1604–07 (2015).
. A recent paper, however, by Cesare Righi and Timothy Simcoe documents evidence of within-technology-group assignments based on sub-technology specializations. See generally Cesare Righi & Timothy Simcoe, Patent Examiner Specialization (Nat’l Bureau of Econ. Research, Working Paper No. 23913, 2017), http://www.nber.org/papers/w23913 (documenting the evidence of sub-technology specialization assignments). However, Righi and Simcoe’s analysis finds no evidence to suggest that applications are sorted across examiners based on the importance or claim breadth of the applications or on their patent worthiness. Id.
. Of course, the validity of a granted patent may also be reviewed by a federal court during a patent infringement or declaratory judgement action. However, there are several reasons why a reversal by PTAB is likely more salient to a patent examiner than a reversal by a federal court. Perhaps most importantly, the average time delay between the issuance of a patent and the final validity decision, which is 8.6 years, is greater than the tenure of a notable portion of patent examiners. John R. Allison & Mark A. Lemley, Empirical Evidence on the Validity of Litigated Patents, 26 AIPLA Q.J. 185, 236 tbl.12 (1998). The time delay between the denial of a patent and a final validity decision by PTAB is far shorter. Id. at 236 tbl.13.
. See supra Section III.A.
. More specifically, to form the necessary risk adjustments, we start by taking the individual sample of applications and regressing the incidence of the application being granted on a set of examiner fixed effects, along with a series of other variables: technology-group fixed effects, GS pay-scale fixed effects, experience group (in year) fixed effects, application-disposition-year fixed effects, and an indicator variable representing whether or not the relevant applicant has “small entity” status. We then take the predicted values of the examiner fixed effects and use these values to indicate the examiner’s risk-adjusted career granting tendencies.
. Frakes & Wasserman, supra note 14, at 550.
. Id. at 554–55.
. We also control for whether or not the applicant has “small entity” status with the Patent Office. Each regression also includes a set of examiner fixed effects, allowing us to account flexibly for fixed differences in inherent granting tendencies across each examiner in the sample. In this light, the graphs presented below can be seen as tracing out an individual examiner’s granting trends as that examiner approaches and moves beyond their first PTAB evaluation event. Examiners that never experienced the relevant PTAB event—e.g., that never had a rejection reversed by PTAB—are also included in the underlying regression. Their presence in the regression is nonetheless useful to help identify the other key parameters of the regression model—e.g., the overall time trends. We also note that the presented results are virtually identical if we also include technology group fixed effects.
. See supra notes 39–40 and accompanying text.
. USPTO PAR 2017, supra note 1, at 181–82. The difference in numbers stems from a number of factors including PTAB’s procedures associated with reviewing a patent grant are more formal and court like, resulting in substantially higher legal fees, than the procedures associated with the Board’s review of patent denials. See Wasserman, The PTO’s Asymmetric Incentives, supra note 32, at 401–06.
. See USPTO PAR 2013, supra note 40, at 23.
. Vishnubhakat et al., supra note 42, at 69.
. See Am. Intellectual Prop. Law Ass’n, Report of the Economic Survey 2017, at 30, 51 (2017) (noting the median cost of post-grant proceedings before the Patent Office to each side is $200,000 through the end of motion practice, $250,000 through PTAB hearing, and $350,000 through appeal whereas the median cost of patent denial proceeding before the Patent Office is less than $10,000).
. The Patent Office sampling percentage has varied over the years. For instance, in 1996 the Agency cut the personnel of the Office of Patent Quality Review in half which resulted in the Agency sampling only two percent of allowed applications, which was well below the four percent sampling rate the Agency had determined was necessary to provide valid results. Office of Inspector Gen., U.S. Dep’t of Commerce, Audit Report No. PTD-9977-7-0001, Patent Quality Controls Are Inadequate (1997).
. See supra Part III.
. U.S. Patent & Trademark Office, Manual of Patent Examining Procedure
§ 1207.01 (2018), https://www.uspto.gov/web/offices/pac/mpep/s1207.html (noting that “[t]he participants of the appeal conference should include (1) the examiner charged with preparation of the examiner’s answer, (2) a supervisory patent examiner (SPE), and (3) another examiner, known as a conferee, having sufficient experience to be of assistance in the consideration of the merits of the issues on appeal”).
. Post Grant Outcomes Pilot, U.S. Patent & Trademark Office, https://www.uspto.gov/
patent/initiatives/post-grant-outcomes-pilot (last visited Apr. 27, 2019).