Rigorous Policy Pilots the USPTO Could Try

The Constitution vests in Congress the authority to “promote the Progress of Science and useful Arts.”1 This clause serves as the basis for the patent system and the issuance of patents by the United States Patent and Trademark Office (“USPTO”).2 The grant of a patent, which bestows to its holder the right to exclude others from practicing the invention for a period of time, does not happen automatically, but only after a process of contestation and negotiation (called “patent prosecution”) between the inventor and the Patent Office. A companion article, Rigorous Policy Pilots, lays out a framework with the shorthand, “MATTER”,3 for proposing policy experiments for agencies to address important open questions in the administration of policy and the law.4 This appendix uses that framework to propose several pilots that the USPTO could implement to address some of the patent system’s most enduring challenges: ensuring both that patents granted by the Office are of high quality, and ensuring the full participation of US innovators in inventing (“inclusive innovation”).

The “MATTER” framework, a distillation of best practices in agency experimentation, includes several steps. First, choose a question or issue that matters or is otherwise a priority on the agency’s learning agenda.5 Next, attend to issues of agency authority and resources and work within or obtain, the authority required. To further the agency’s learning agenda, articulate a theory of change in connection with the proposed policy or rule change and identify a rigorous strategy for testing the theory. In doing so, take care to specify the relevant evidence, for example, regarding whether the policy change had its expected impact, and a strategy for collecting the evidence, with adequate resources. The Parts below apply each step of this framework to propose pilots the USPTO could try for advancing patent quality and inclusive innovation.

I.     Issues that Matter at the USPTO: Patent Quality and Inclusion in Innovation

The basic function of the USPTO is to determine whether an invention truly represents “[p]rogress . . . of the useful arts.”6 The progress requirement is enshrined in at least two statutory provisions: novelty (35 U.S.C. § 102) and non-obviousness (35 U.S.C. §103).7 But telling whether an invention is new and nonobvious over the “prior art” is hard. Even Thomas Jefferson, one of the nation’s first patent examiners, struggled with it, writing in a letter to Isaac McPherson, “I know well the difficulty of drawing a line between the things which are worth to the public the embarrassment of an exclusive patent, and those which are not.”8 The task proved so difficult for him and the other members of the panel tasked with reviewing patent applications that given competing demands on their time, robust examination was eliminated and replaced with a system of registration in 1793.9

Over the past decade, patent quality, particularly among software and business method patents, has attracted intense policy attention, and not the good kind. Justice Kennedy’s concurrence in the eBay decision blamed the “potential vagueness and suspect validity of [business method] patents”10 In 2011, Congress created three new ways to challenge the validity of patents through reviews by administrative law judges at the USPTO, including immediately after they have been granted.11 81 percent of the patent claims finally reviewed (as opposed to challenged) by the Patent Trial and Appeal Board (“PTAB”) through inter partes review, the most popular of these three methods, have been partially or fully invalidated,12 leading the former chief judge of the Federal Circuit to call the Board a “death squad.”13

While the USPTO has launched several initiatives to improve patent quality, their effectiveness has been limited by a lack of a way to measure patent quality. In Part III below, Patent Quality Pilots, I suggest several quality metrics based on natural experiments, natural yardsticks, and independent or synthetic assessments.14 I then argue for their inclusion in pilots to enhance the robustness of patent vetting and flexibly allocate examination resources in close cases.

A second area that could benefit from rigorous piloting is inclusive innovation. From the start, the U.S. patent system has included distinct features meant to encourage participation and inclusion15 in inventing. The first patent system featured relatively low fees,16 the ability to accept applications by mail,17 and a merits-rather than patronage-based system for awarding patents, to support low-income, rural, and worthy inventors. A commitment to inclusive inventing also led Congress, in 1982, to introduce fee discounts for small, non-profit, and individual inventors.18 In 2011, as part of the America Invents Act (“AIA”), Congress created a new tier of fees for the smallest “micro-entity” inventors,19 and created regional offices of the USPTO in Detroit, Dallas, Denver, and San Jose to offer services across the country, not just in Alexandria, VA, where the USPTO is headquartered.

Women comprise only 12 percent of the inventors that apply for patents,20 a share that is substantially lower than the share of women among STEM degree holders and in the STEM workforce.21 While comparable point estimates for the representation of non-binary individuals as well as underrepresented minorities are not available due to the lack of reliable methods of identification,22 Raj Chetty and his colleagues have concluded that, “[i]f women, minorities, and children from low-income families were to invent at the same rate as white men from high-income families, there would be four times as many inventors in America as there are today.”23 The inclusion of underrepresented groups in the patent system has emerged as a recent Congressional priority. In 2011, Congress charged the USPTO with “‘establish[ing] methods for studying the diversity of patent applicants, including those applicants who are minorities, women, or veterans.”24 The recently enacted SUCCESS Act of 2018 included a sense of Congress that, “the United States has the responsibility to work with the private sector to close the gap in the number of patents applied for and obtained by women and minorities to harness the maximum innovative potential and continue to promote United States leadership in the global economy.”25

Rigorously tracking the maturity of applications to patents can yield useful insights for advancing equity in innovation. In Part IV, Pilots to Increase Inclusion in Innovation, I draw from existing and new analyses to explore opportunities for increasing participation in the patent system. I propose two pilots: one with the goal of reducing the attrition of small and micro-entity applications by using technology to highlight § 112 defects, and another for testing for the possibility of implicit gender bias in the examination of patents.

II.     Authority and Resources to Pilot At The USPTO

The USPTO has a number of attributes that position it well for piloting: a strong open data infrastructure with few privacy encumbrances, a decentralized structure, and a large number of examiners and examination transactions with which to test different interventions. As an innovation agency that generates its own fees, it also has a less politicized mandate as well as a strong culture of piloting. As such, the paragraphs below discuss adding rigor to existing pilots in the patent quality area, and using mechanism and other pilots to encourage greater inclusion in the patent system.

As discussed in Part II of Rigorous Policy Pilots,26 a policy pilot’s design, whether based, for example, on case studies, changes to agency procedure or law, or changes that will impact applicant behavior—will impact its legality. The proposed pilots below follow the template of previous USPTO quality pilots that vary the behavior of examiners, and should pose few legal obstacles and, in some cases, may not even necessarily need notice and comment. Practical hurdles with piloting, as well as more significant obstacles associated with scaling proposed policies, which could implicate cost and timing of patenting and require negotiations with the union, are likely to be more significant. But rigorously piloting can build a strong evidentiary base from which to determine whether significant changes are justified in the first place.

Depending on how they are implemented, the patent inclusion pilots discussed below could potentially involve varying the behavior of applicants. A pilot that contemplates providing benefits to inventors based on their economic status, rather than on the basis of protected class status, is unlikely to implicate deprivations of protected interests. Though the problem of gender bias implicates a suspect class, the test proposed below of considering the impact of examiner behavior based on the distinction between male and female sounding names, would not. Because it represents a test for implicit bias, the proposed test functions as a diagnostic of possible differential treatment rather representing differential treatment itself.

Regardless, the pilots discussed below would squarely fit into the USPTO’s implementation of its core responsibility of establishing regulations that “govern the conduct of proceedings in the Office.”27 Structured properly, the quality pilots discussed should easily be found to further the agency’s stated top objective, to “optimize patent quality and timeliness.”28 The inclusion pilots described, which aim to level the playing field for underrepresented inventors, would also likely be found by a court to further “the public interest in continuing to safeguard broad access to the United States patent system,” as Congress articulated in the 2011 America Invents Act.29 But while the concept of piloting, and even doing so to promote patent quality, is not new, implementing pilots through randomized controlled trials would represent a departure from standard practice. Below I explore how to do so and why it may be worthwhile.

III.     Patent Quality Pilots

A.     Mastering Patent Quality Metrics (Evidence)

As discussed above, one challenge with patent quality is a lack of a way to measure it. Building on the discussion of measurement in Part III in the companion article,30 the paragraphs below outline comparative metrics for measuring quality based on (1) natural experiments, created when the same invention is filed in another, reference jurisdiction, creating a “twin review”; (2) natural yardsticks, created when a granted application is reviewed, for example by an internal review board of the agency, the Patent Trials and Appeals Board or a court, spurring “second look review”; (3) independent assessments by expert human or technical-systems; and (4) process metrics over time.

Twin Review: As I and others have previously exploited, when a patent seeker submits an application to the U.S. Patent Office, the same application may be submitted to another patent office, such as the European Patent Office (“EPO”),31 for examination by both. This creates a natural experiment involving “identical twin” applications, each examined by multiple offices applying similar patentability standards. Although the patents that have counterpart twins tend to be more valuable than those that do not, comparisons between twins can provide useful, comparative information. The EPO (or, possibly, other jurisdiction) examination can provide a control of sorts, or at least benchmark, for the US examination process, though care needs to be taken to reduce other sources of variation.32

Second Look Review: Patent disputes often include challenges to patents for their legal sufficiency and compliance with the statutory patentability criteria. The review, whether carried out by the Patent Trials and Appeals Board or a court, for example, comprises a second, closer look, at the patent and underlying invention following the “first look” of the patent examiner. The scope of “second look” review depends on the legal procedure used to challenge the patent. While inter partes review processes permit reconsideration of an application’s compliance with the novelty and non-obviousness requirements of the patent law, “covered business method reviews” and post grant review allow challenges on other grounds.33 As such, the PTAB’s assessment of a patent provides a natural yardstick for comparison with USPTO examination, since the PTAB’s three-judge panels are essentially repeating the work of the USPTO examiner, albeit with more resources (three administrative judges and the input of a third party), less autonomy (since references are provided by the challenger) and more time.

Independent Assessment: While twin and second look reviews can be useful, especially retrospectively, not all patents are subject to them. Further, the lapse in time between when a patent is examined and then vetted through these other processes can, like litigation, make interventions impossible to evaluate within a reasonable timeframe. A few other sources of information can be the source of additional, independent assessments.

First, for benchmarking the search for relevant prior art, the applicant’s own sense of what references are relevant provides one datapoint. That is because applicants have a duty to disclose relevant references of which they are aware to the patent office pursuant to Rule 56,34 through an “information disclosure statement” (“IDS”). The incentive to submit relevant prior art when a claim is pending is high because failure to do so can form the basis of disciplinary proceedings for the patent attorney. It can also, potentially, lead to the unenforceability of the patent pursuant to an equitable “unclean hands” defense as articulated by the Supreme Court in Precision Instrument Manufacturing Co. v. Automotive Maintenance Machinery Co.35

Second, for benchmarking the quality of examination more generally, synthetic, expert evaluations of quality could be used. Depending on the treatment to be tested, such experts36 could be asked to evaluate applications with and without the treatment, and the differences in outcomes, compared. A recent study of decisions regarding statutory provision 35 U.S.C. § 101 involving 231 prosecutors found a high-level of agreement between the prosecutors and courts, suggesting that prosecutors could be used to predict court outcomes.37 The use of artificial intelligence or automated systems that have transparent and clear evaluation criteria, as discussed briefly below, would also be worth exploring. Data from the USPTO’s own quality-control audits, carried out by the Office of Patent Quality Assurance, could also be leveraged. Whatever approach or combination of approaches is used, the methodology for evaluation should be transparent and clear, in order to build trust in its assessment.

Metrics Over Time: Finally, depending on the quality intervention to be tested and how the intervention is deployed, control groups over time can be identified. Differences in differences approaches can potentially be applied to quality pilots directed at specific classes of applicants, for example small entities, applicants from certain technology areas, or applicants from a certain geography. Depending on how changes are introduced, other quasi-experimental techniques like “before and after” and regression discontinuity may also be appropriate.

It is important to note that each of these metrics has its limitations. Even when foreign and US patent examination outcomes differ, differences in applicant behavior, the law or its administration, or, over time, relevant changes to factors other than differences in examination approach may explain observed differences. The patents that are challenged in the PTAB and in litigation are highly selected, and not representative of “average” patents. Applicant IDS’ typically reflect a variety of considerations and embed differences in industry practice that need to be taken into account. The dynamic nature of the patent system also means that metrics over time need to be generated and evaluated with caution. For all of these reasons, a “mixed metrics,” as well as “mixed methods” approach makes sense.

B.     Increasing the Robustness of Patent Vetting

Only truly novel and non-obvious inventions deserve patents. But ensuring that examiners consider the complete spectrum of possible prior art has long been recognized to be impractical, “enough” prior art is a more realistic goal. In 1967, a Presidential Commission opposed granting software patents due to prior art challenges, stating that “[t]he Patent Office now cannot examine applications for programs because of the lack of a classification technique and the requisite search files. Even if these were available, reliable searches would not be feasible or economic because of the tremendous volume of prior art being generated.”38 In Microsoft Corp. v. i4i Ltd. Partnership, the Supreme Court acknowledged what is widely known within the patent community, that patent examiners cannot and do not consider all relevant prior art during examination.39 When asked to reconsider the presumption of validity that the statute gives to patents,40 particularly in the face of prior art never before the Patent Office, the Court stated that, “the jury may be instructed to consider that it has heard evidence that the PTO had no opportunity to evaluate before granting the patent.”41 In doing so it acknowledged what numerous appellate courts have previously acknowledged, that there will be cases where “the evidence in an infringement action was never considered by the PTO.”42

Most of the time, examiners tend to cite the references that are most familiar to them—other patents.43 This does not necessarily make patents different than other types of documents—consider the ways in which law review articles tend to cite other law review articles, and economics papers cite other economics papers. However, the task of a patent examiner is fundamentally different than that of academic authors because to determine a patent requires vetting it in light of all statutory forms of prior art, patent or non-patent. This presents the impossible task of keeping up on all types of literature and sources of information. This is particularly crucial in fields in which many publicly reported advances first appear and sometimes only appear, not in a patent, but a public repository.44

1.     Evidence of a Robustness Gap

Access to relevant prior art has a long history of being a priority for the USPTO.45 But despite many efforts to enhance access to prior art, the USPTO has not, to date, rigorously measured whether or not this has resulted in more robust vetting of applications. This subsection proposes that the USPTO do so, by adopting quality metrics associated with the examiner use of non-patent literature, for several reasons. First, it is generally undisputed that, “[c]ore to substantive quality is the prior art search.”46 Second, the extent of US examiner citation of non-patent literature citation can also reliably be approximated at scale using methods previously developed and tested47 and machine learning based coding of office actions carried out by the USPTO, lowering the cost of tracking this metric. Finally, the stakes are high. Though receiving relatively less attention than topics like patentable subject matter, prior art issues are part of at least 79 percent of all office actions, and factor into almost every single patent examined.48 Further, “the ever-accelerating publication and accessibility explosions” in prior art recently acknowledged by USPTO Director Andrei Iancu mean that the problem is not going away on its own.49

But even if examiner’s reliance on non-patent literature citation is trackable, how can we determine whether or not this is an area in which the USPTO needs improvement? In earlier work, I used the natural experiments approach described above50 to document that, among patent applications filed in 2002, the rate of USPTO examiner citation of non-patent literature (“NPL”) when examining patents was not only low, but lower than that of the European Patent Office. This finding was robust across multiple measures of NPL citation51 and in every technology sector (except chemistry as to one measure). Across fields, four percent of the art relied upon by a U.S. Examiner was NPL, while the comparable rate in the EPO was 20 percent.52

Building on this work, below I present a “comparative” analysis that considers the extent to which each of four different assessors cited NPL in their evaluation of the prior art relevant to a particular invention. To do so, I chose patents that had been invalidated by the PTAB through the inter partes review process,53 as they essentially present disagreements between the examiner, which deemed the invention patentable in view of the prior art, and the PTAB, a panel of expert administrative law judges, which came to the opposite conclusion. In Figure 1, I also add the NPL citation rate of the applicant themselves, as provided through an information disclosure statement.

The gaps appear to be stark. Among patents invalidated in IPR (N=906), US examiners cited non-patent literature 13 percent of the time in their search reports.54 But when the PTAB made its decision that claims in these same patents were invalid, it cited NPL 41 percent of the time.55 Applicants cited NPL even more frequently, 66 percent of the time.56 When the same metrics were calculated only among those patents that had, as applications, also been the subject of European examination (N=240), the rates of NPL citation were 14 percent (US examiner), 34 percent (EP examiner), 45 percent (PTAB invalidation decision) and 72 percent (applicant IDS).57 The large gaps in NPL citation between examiners and applicants (IDS), and examiner and PTAB held across patent technology centers (N=73-186): that is to say, across every USPTO technology center, among invalidated patents, the rate of US examiner NPL citation was lower than the rate of applicant NPL citation (in IDS) and the rate of NPL citation by the PTAB.58


Source: Author’s analysis.

Of course, correlation does not imply causation, and it cannot be assumed from these contrasts that a lack of NPL citation is the sole factor responsible for the issuance of patents later proved to be invalid.59 However, because the inter partes review inquiry considers only the patent’s validity in light of the prior art, it squarely presents the question, were there references that the examiner missed when examining the patent application? There is some evidence suggesting that the robustness of US examiner vetting, relative to EP examiner vetting, may have implications for outcomes. Among the US patents granted by the USPTO but fully invalidated in IPR, 202 had an EPO counterpart application. Of these, approximately 1/3 of the EPO applications never matured into patents, primarily because they were withdrawn or revoked.60 But while similar applications that did not proceed in the EPO but were granted in the US were 30 percent more likely to have NPL cited in the search report than their US counterpart,61 the difference among cases where both offices granted the patent was much smaller, a third of that or 10 percent.62

2.     Testing Timing and Time-Based Theories of Change

Rather than propose a new pilot, this Essay proposes that past, existing and future prior art initiatives be rigorously evaluated, at least in part, based on a new metric—the extent to which examiners considered a diversity of prior art sources, including non-patent literature. Retrospectively, this could be accomplished by looking for natural experiments among the approaches to prior art that have been tried through pilots. Prospectively, this would most easily be accomplished by introducing randomization in the rollout of each new agency rule or regulation and by tracking and evaluating examiner-cited NPL rates. If the USPTO decided to vary examiner behavior in the vein of previous pilots, it could follow the normal notice-and-comment process to ensure Administrative Procedures Act (“APA”) compliance. Randomization of examination approaches to prior art could support mechanism testing to ascertain whether, for example, enhanced access to search tools or search results are actually likely to lead to more consideration of prior art sources.

Why wouldn’t a patent examiner consider all relevant prior art? The answer that Thomas Jefferson might give is that doing so is basically an impossible task, given the press of time and the amount of potential prior art. However, if the question is, why are US examiners citing non-patent literature less than the EP and PTAB, there may be more specific reasons. One distinct feature of US examination as compared to PTAB and EP review is the gradual nature of prior art discovery. In the EP, search and examination are bifurcated and so a thorough search is done upfront. Likewise, during PTAB proceedings, all of the art is provided at the beginning. This is not necessarily the case with examination at the USPTO, as the examiner refreshes the search as the patent claims evolve over the prosecution. Another difference has to do with the extent to which prior art is provided by the applicant. In examination, the applicant may furnish prior art to the Patent Office for its consideration, through an information disclosure statement submission. The presence of applicant-submitted art might create time pressures on examiners. There are no provisions for applicant submissions of art in the EP or PTAB.63 Other factors contributing to what art does and does not get considered might include an examiner’s habits, familiarity with different databases, and seniority level.

The impact of these factors could be probed using retrospective or prospective pilots using measures of robust search including examiner-citation of non-patent literature. For example, a “search-first” pilot could mimic European style examination and require the examiner to search the entire application upfront, and be bound by this initial, comprehensive search. Consistent with similar “compact prosecution” initiatives that aim to get the best prior art before the examiner early in the examination process, this would have the benefit of providing an “early signal” to patent applicants. Equipped with the information about what the universe of relevant art looks like, the applicant can make an “early decision” based on full information regarding whether to pursue the application.

A more modest idea would be to test one of the assumptions that underlies a number of the USPTO’s previous initiatives, that earlier access to prior art will lead to more robust vetting. Initiatives like the Access to Relevant Prior Art initiative are premised on the importance of “get[ting] the best prior art in front of an examiner as soon as possible in the examination process.”64 However, providing examiners prior art early, by itself, may not cause them to rely on a more robust range of references, as a number of factors other than availability impact the use of prior art by examiners. Previous work by Cotropia and his co-authors suggests, for example, that examiners have a strong preference for citing references that they themselves find, rather than references provided by applicants, which require time to digest and then apply.65 Whether or not earlier access to references leads to more consideration of NPL could provide one way to test whether timing by itself will lead to consideration of a diversity of sources.

Another policy “treatment” to test would be the combination of timing and time. For example, in inter partes review proceedings before the PTAB, the Board receives references from the applicant, but it also receives a description of the relevance of the references. Because the applicant has already explained the relevance of the reference in its submission, new NPL references are not as (time-) costly for the PTAB to incorporate in its evaluation. Likewise, obtaining foreign search reports that designate how a reference is important early in the process provide the US examiner not only with early information about key references, but information that is augmented with insights about the importance of the references. The curation and context, and not just the raw references, can help examiners vet patents more efficiently.

Identifying natural experiments based on past pilots, making adjustments to current pilots, or launching new pilots with randomization, then measuring their impact on considered prior art can help identify ways of improving the robustness of patent vetting. Consideration of not only the first-order metric of examiner-citation of prior art,66 but also second-order, outcome measures like the time to resolution and grant rate can elucidate not only the mechanisms by which vetting can be improved but also the payoff associated therewith.

C.     “Team/Time on Demand” Examination

Another explanation for the patents of “suspect quality” of Kennedy’s eBay concurrence is inconsistency of examination.67 As Mark Lemley and then Professor, now-Federal Circuit Judge Kimberly Moore describe in an influential article, the uneven quality of patent examination means that undeserving inventions will slip through the application process, as long as the applicant is patient and “wears down the examiner.”68 “There may be as many patent offices as patent examiners,”69 it has been said. When a junior patent examiner evaluates a patent, the examiner must get a supervisor or another examiner with full signatory authority to review and sign off on the work. A senior examiner with full signatory authority (GS-14 and above), in contrast, generally works independently and with less time. In between are examiners with partial signatory authority.70 A number of studies suggest that a patent’s quality is inversely correlated with the examiner’s seniority. Michael Frakes and Melissa Wasserman have found that examiners tend to increase their grant rates as they become more senior,71 and Brian Love and his co‑authors have found that patents assigned to experienced examiners are less likely to survive inter partes review.72 Marco and Miller have found that patents issued by senior examiners are more likely to be litigated.73

Spurred in part by these studies, increased attention has been paid recently to the amount of time allotted to senior examiners.74 70 percent of patent examiner respondents to a survey by the GAO said they did not have enough time to complete a thorough examination given a typical workload.75 However, though receiving less attention, another factor distinguishing senior from junior examiners is team work. Junior examiners are supervised and therefore have the benefit of team examination. But, the more senior an examiner gets, the more likely that she is working alone, as shown in Figure 3 below.76

Figure 3: USPTO Examination Teams


Source: USPTO Office Action Dataset, author’s analysis

1.     Looking Back at Second Pair of Eyes Review

Sensitivity to time constraints has previously led USPTO leadership to increase the time allocated to examiners.77 However, more time in examination—all other things being equal—leads to a longer backlog and pendency. Perhaps this explains in part why providing more time to all senior examiners to shore up patent quality has not yet been pursued by the USPTO. However, it is notable that the USPTO has selectively provided extra time in certain contexts and for specific reasons through “Second Pair of Eyes Review” (“SPER”) programs. From around 2000 to 2011, the SPER programs required class 705 business-method patent applications to be evaluated by two or more examiners before being allowed.78 In 2003, the USPTO announced the application of SPER review to restriction decisions in biotechnology patent applications in order to enhance consistency.79 A retrospective analysis pilot based on the identification of natural experiments could ask the question —how did the additional time impact intra-art unit consistency? Were issued patents of higher quality, based on the benchmarks discussed earlier?

2.     Looking Forward at Team/Time on Demand

Extra time allocated to examiners to ensure high quality is a limited resource whose allocation can be evaluated using random distribution. In fact, as described earlier, randomized distribution of scarce benefits is a well-recognized way of resolving the twin dilemmas posed by unequal treatment in experimentation and the real-world constraints of finite budgets. Building upon the USPTO’s efforts and the teaming model that is used among applications evaluated by junior examiners and in the EPO,80 this Essay proposes consideration of a new “Examination Team or Time on Demand” pilot.

Like previous efforts, a “team/time on demand” pilot would provide additional time to cases that are likely to present challenges. But, unlike programs such as SPER which implemented a single, top-down decision to allocate additional resources to all applications that met a certain profile, a team/time on demand pilot would make the decision of when to apply additional resources a local one, determined at the art unit level by individual examiners who would themselves decide when extra hours or a second opinion is needed. Examiners could allocate the extra resource in the way they desired, either by themselves using the additional time or by choosing to partner with other examiners who, for example, are expert or trusted. The extra time would be allocated accordingly.

Such an experimental design would reflect the institutional wisdom embedded in previous SPER programs as well as EPO Examination that having two (or more) examiners agree on examination can increase quality and consistency. However, the design would also allow the resource decision to be informed by local examiner “know-how” rather than a top-down decision, not only about what cases need extra time or an extra pair of eyes, but also regarding who on the examination team might provide this extra help. The resource could be applied dynamically only where needed, rather than allocated in a fixed manner to all cases.

If successful, team/time examination on demand would enhance patent quality and consistency without substantially raising the average cost of examination. Thus, the pilots could include several first-order metrics, including intra-art unit consistency and robustness of vetting as well as cost and overall higher quality, based on the benchmarks discussed earlier.

IV.     Pilots To Enhance Inclusion In Innovation

A.     Supporting Small Inventors

As discussed above, improving inclusion in innovation has long been a Congressional and national economic priority. Reducing the cost barriers to participation has been one strategy for doing so. For some time, a 50 percent discount off of fees paid to the Patent Office has been available to help small filers, including for-profit firms with less than 500 employees, independent inventors, and nonprofits.81 Starting in early 2013, filers that fall below a certain revenue and filing thresholds have been entitled to even deeper “microentity” discounts of 75 percent.82 In addition to helping prevent small inventors from meeting the fate of Charles Dickens’ Poor Man, a fictitious patent applicant driven to ruin by the onerousness and expense of the patenting process,83 these discounts provide a thus-far underused identification strategy for identifying a patent owner’s size that is based on the fees it pays. Every time an eligible applicant pays a fee to the USPTO—at the point of filing, prosecution, and then maintenance—thus presents an opportunity to observe its size.

1.     The Small Entity Gap in Obtaining Patents

A previous analysis determined, based on payment records, the prevalence of small (including “small” and “micro”) entity filings at the USPTO as compared to large entity filings,84 as well as the share of independent inventors. The data reveal some troubling trends for those who are interested in bolstering the participation of small inventors in the patent system. Not only have shares of small entities fallen over time, but within patent applications, there has been a successive decrease in small entity shares across the lifetime of a patent, implying that small entities are dropping out at a heightened rate at every phase of patenting. For example, 33 percent of patent applications filed in 2000 were by small entities, but by the time of patent issuance, that share had shrunk to 25 percent, and to 23 percent, 18 percent, and 15 percent at the milestones of first, second, and third maintenance fees, as shown in Figure 4 below. This trend was observed across all the periods studied.85 If Congress is encouraging more inventors to file with lower fees, but this just leads to them dropping out before the patent issues, the applicant will have gained nothing in the long-term but lost time and attorney, USPTO and other possible fees associated with the patent’s filings as a result of their efforts. Small applicants will be worse off than if they had never even tried to file, achieving the exact opposite of the law change’s intended impact.

Figure 4: Discounted Entity Shares Over the Patent Lifecycle


Source: Author’s analysis86

2.     Strengthening Patent Disclosure

Why are small entities dropping out even before their patents issue? The analysis described above suggests that one culprit is the sufficiency of submitted applications under 35 U.S.C. § 112, which requires that an invention be adequately disclosed, precisely claimed, and that the application clearly point out and distinctly claim the subject matter of the invention.87 Problems can also arise when claim language or references are unclear and the reader becomes uncertain about the drafter’s intent, thus violating the so-called “definiteness” requirement.88 For example, a claim that refers to “said aluminum box,” but recites only “a box” earlier in the claim, lacks clarity as to whether the two are referring to the same box.89 Similarly, the phrase “R is halogen, for example, chlorine” fails the definiteness test because it’s unclear what is meant.90

Other errors may be introduced by a lack of correspondence between the claims and the text of the specification, the figures and the text, or numbering problems in the claims. Specifications that fail to support the claims can drive hours of work and rework on the part of examiners and applicants because it is unclear exactly what the invention is. In the case of insufficient disclosure, this lack of support can be fatal for the application which, despite representing great effort and expenditure, may never mature into a patent. 35 U.S.C. § 112(a) requires a specification to contain a written description of the invention, and therefore tie the claims and the disclosure together, but the connection can be lacking. Indeed, differences between the claims and the inventive concept, and differences between the claims and the disclosure in the specification have been ranked by examiners to be the top-ranked problems, respectively, among incoming patent applications.91

An analysis of the relative prevalence of § 112(b) errors and objections among the office actions of large (undiscounted), small, and micro entity applicants suggest that the quality of underlying applications differs by owner size. Across different technology types and times (from 2000 to 2017), the error rate appears to systematically increase as the size of the owner decreases. In total, 26 percent, 36 percent, and 43 percent of large, small, and micro entity office action respectively included a § 112(b) rejection, as shown in Figure 5A below.92 Likewise, errors in office actions, as proxied by objections in applications, were more prevalent among small and micro-entities, as shown in Figure 5B.

Figure 5A: 112(b) Rejections in Office Actions


Figure 5B: Objections in Office Actions


Source: Author’s analysis using the USPTO Office Action Dataset and USPTO Data

3.     Strengthening Compliance with 112 Through Error Correction Technology

How can the dropout effect among small entities be reduced? Currently, a number of commercial products exist to help applicants detect errors and point out weaknesses in applications. For example, TurboPatents’s “RoboReview” product uses artificial intelligence techniques to identify potential § 112(a) issues (unsupported claim terms), § 112(b) errors, and related ministerial issues (antecedent basis, figure reference and claim order and format issues).93 ClaimMaster offers a similar service.94 However, under-resourced and less sophisticated patent applicants are among the least likely to have access to fee-based tools, particularly when they represent themselves pro se.

One approach the USPTO could pilot is to make available existing or new (for example, developed by or for the USPTO) error detection tools available to discounted or all applicants to alert them of deficiencies in their applications and suggest corrections prior to filing. The use of tools by executive agencies to reduce errors in public submissions is precedented.95 The Internal Revenue Service (“IRS”), for example, has adopted barcode technology to facilitate the consistent reporting of data about the filer. When filling out IRS forms, a person may report information in a paper form and then need to reuse that data in another form, risking transcription or related errors. In 2003, at the initiative of several employees in the Small Business Self-Employed (“SBSE”) operating division of the IRS, the agency launched an experimental pilot based on the Schedule K-1 family of forms because of its high volume of paper submissions.96 The IRS partnered with the tax software development community to develop the appropriate technology.97 During the pilot, eight percent of all of the K-1’s received used bar-coded data, reducing the risk of error in about 1.5 million forms.98 Building on this pilot, the IRS formally introduced bar code technology for an expanded number of forms.99

In the case of an error correction pilot, the USPTO could start by working with technology providers to provide access to vetted tools to detect problems within applications and suggest corrections and test their openness to tools. The agency could then offer the tool to a sample of discounted applicants and then randomize the treatment, or recruit firms to participate in a pilot, taking care to ensure that the control and treated groups were comparable. If the desire is to measure the impact of access to technology on the quality of applications, the key would be ensuring that the distribution of the offer or delivery of tech tools was truly random. Uptake rates and objection and rejection rates of the treated and control groups could then be compared. If the intervention succeeded, applicants would detect and correct their errors when prompted by the tool before submitting them to the PTO, resulting in fewer objections and § 112 rejections. Outcomes to test, then, could include whether or not the technology reduced error rates, and whether it also led to a higher likelihood of patent issuance, and also whether it led to the closing of the gaps between small and large entities.

B.     Finding Our Lost Marie Curies

Another context in which policy pilots could be used to advance inclusive innovation is in the promotion of patenting by women, as is called for in the 2018 SUCCESS Act. Women make up about half of the college-educated U.S. workforce, but only 24 percent of the STEM workforce.100 While women earn nearly three out of ten STEM degrees,101 they represent less than half that share, 12 percent, of inventors on U.S.-origin patents.102 Unequal access and participation of women and other underrepresented minorities in innovation contributes to an “earnings, income, employment, and wealth gap,” Lisa Cook has observed.103

The reasons for the extreme under-representation of women in patenting are complex and appear to stem, in part, from selection out of less patent-intensive fields and patent-intensive job tasks.104 As to the gender gap in STEM in general, numerous studies have found the differences in math and science aptitude to be “small or nonexistent,”105 drastically reducing the chances that differences in ability are the primary explanation.106 However, the relative contributions of other factors, including lifestyle and personal choices (which still may reflect structural inequalities) and gender discrimination are hard to tease apart because many existing studies are based on observational data and are correlational.107

1.     The Gender Gap in Obtaining Patents

While many of the factors potentially contributing to the gender gap in inventing are outside the patent system, some are squarely within it. Structural barriers to the full participation of women inventors date back to the earliest inventions. For example, Hannah Wilkinson Slater is credited by many as the first woman to hold a patent, on a method for producing cotton-sewing thread.108 Her invention post-dated the first grant to a man, by Samuel Hopkins, over potash, an ingredient in fertilizer, by three years.109 But she was not given independent credit for the invention—the Patent Office issued the patent to Mrs. Samuel Slater.110 The rights of married women to patent independently of their husbands was only confirmed about a century after that.111

Female inventors have a lower success rate on applications to the US Patent Office than male inventors, a striking study by Kyle Jensen and his colleagues has documented.112 All-women inventor teams were 21 percent less likely to have their patents awarded than all-male teams, a difference that shrank to seven percent when controlling for technology (by technology classification).113 The patents awarded to women were worse (had fewer independent claims and had more words added, making them narrower) than the ones awarded to men and were less likely to be maintained.114

Patenting is not the only innovation context in which applications from women do worse. Among job applications that differed only with respect to the gender of the applicant (as identifiable by name), causal work has found that science professors, both male and female, favor male students.115 In the context of grant funding, a crucial milestone for many fields of academia, studies across countries and disciplines show that male researchers receive more research funding than their female peers.116 At the National Institutes of Health (“NIH”), a governmental agency like the USPTO, for example, awards to first-time principal investigators are 25 percent lower to women than to men with statistically indistinguishable publication records.117 A quasi-experimental study of gender gaps in funding provided by the Canadian Institutes of Health Research found that gender gaps in grant funding stem from women being evaluated less favorably as principal investigators, not from differences in the quality of proposals led by men and women.118

What contributes to gender disparities in patenting? It is important to acknowledge that applicant and Patent Office decisions each contribute in varying degrees to who gets a patent, and which patents remain in force. For example, applicants alone select the inventions on which to seek patents, as well as decide whether or not to keep a patent, once granted, in force. The PTO, for its part, examines applications and sets the fees and the cost of patenting. The high cost of patenting has previously been cited as an obstacle to female patenting,119 and a desire to remove this barrier to participation has motivated the creation of deep discounts for small and micro-entities.

The vast majority of the gap between employment and patenting appears to be attributable to a lack of application, as even if the 12 percent female inventor rate were increased to reflect grant parity using the estimates of Jensen et al., the female inventor rate would only increase by one to three percent,120 leaving more than a double digit gap between female technical employment (28 percent) and patent application (up to 15 percent). But as to the perhaps most important outcome, whether or not a patent was granted, conditional upon application, both examiner and applicant behavior matter: the examiner determines whether or not and how to reject an application, and the applicant decides how to respond, and whether to continue pursuing the patent. This makes it hard to tell if examiners are reviewing applications differently, resulting in an “evaluation” effect, or alternatively, if differences in the effort applied to pursuing a patent application, or a “prosecution effort” effect, is primarily responsible for the observed difference.

How important are evaluation as opposed to prosecution effort effects to the lower success rate of female applications? Two data points point in different directions. First, when the examiner allows an application without consultation with the applicant through a so-called “first action allowance,” a pure “evaluation effect” on grant rates can be observed. It appears that differences in first action allowance rates are relatively small,121 suggesting that if an evaluation effect is contributing to a disparity in grant rates, that it is quite modest, at least at the first action stage.122 Cutting against this finding, however, is the finding that the gender disparity in grant rates appear to be much greater among inventors with familiar names like Jane and Lily, which were associated with an 8 percent gender gap, rather than unfamiliar names like Kunnath, which were associated with a 2.8 percent gender gap.123 As Jensen et al. explains, because the gender of the inventor is presumably known to the applicant as well as inventor, implicit bias in evaluation, based on names, could be operating. Because foreign applicants tend to have forenames that are less readily associated with a particular gender, the operation of such a mechanism could have the unintended consequence of particularly disadvantaging US women inventors.124

2.     An Experimental Approach to Testing for Implicit Bias in Patent Examination

A well-designed mechanism pilot would be worth considering at the USPTO to isolate the impact of gender as opposed to other factors on how applications are evaluated. One relatively straightforward and inexpensive way to do so would be through a mechanism test. Borrowing from “resume studies” that send identical resumes with different names to evaluators, a gender pilot could test the impact of a person’s name on patent examination by sending identical patent applications to examiners, half of which have the female version of a name and half of which have the male version of the name.125 Carrying out a study in which the only difference between the treatment and control is the applicant’s name has the benefit of reducing the impact of unobserved variables.

Rather than subjecting the application to a full examination, to keep costs low, the USPTO might be able to exploit pre-existing processes for assessing the readiness of patents. Each examiner could be given one version of the application and asked to rate it for its compliance with one or more statutory categories of patentability. The application and rating scheme should be designed carefully so as to elicit a spread of ratings, not just a single binary decision. Another approach would be to exploit natural experiments, for example associated with the omission of first names (or inclusion of initials only).126 If differences between applications including female inventors where the first name was and wasn’t disclosed were observed, this could provide further evidence of name-based implicit bias against female inventors. However, contact through other means, for example direct contact with the inventors or patent attorney, however, would need to be ruled out. Uncovering the true gender identity of the applicant would also require successful attempts to locate and confirm the actual gender of the inventor, which could introduce considerable selection bias.

Yet another approach would be to carry out field experiments in coordination with stakeholders willing to blind (or not blind) their applications. Among a pool of otherwise equivalent applications, a random process could be used to select which applications are in the treatment vs. control group. However, there is currently a non-zero risk and cost associated with doing so. The Application Datasheet (“ADS”) rule127 and the inventor’s oath or declaration rule128 require the legal name of the inventor to be supplied. If the inventor’s legal name includes a first name, supplying less than that, for example a first initial or just a blank, would not comply with the rule. Consultations with the USPTO129 have confirmed that if an applicant wanted to initially provide a “blinded” (e.g. first initial or omitted) name followed by a full first name, then a petition to correct the name of the inventor130 would be required, introducing the need for attorney’s fees and a fee to the USPTO,131 as well as uncertainty because the petition would have to be formally granted.

The USPTO could dramatically reduce this risk however, in any number of ways—by creating a non-petition-based pathway to correcting inventorship name, encoding the presumptive grant of a name-correcting petition and waiving its fees, or providing other procedural support. As such the USPTO has several options for affirmatively acting—either through its own pilot or supporting the private sector in its pursuit of such pilot or both—to support the investigation of the 7–21 percent gender gap in inventing.

V.     Conclusion

Every day, governmental agencies are implementing policies and regulations intended to improve the functioning of the administrative state and the lives of citizens. Is it working? Among the many open questions in the administration of policy and the law, I believe this one should remain front and center. The companion article to this short essay, Rigorous Policy Pilots, lays out a framework with the shorthand, “MATTER”, 132 for proposing policy experiments for agencies to address such open questions. This Essay has applied this framework to propose several pilots that the USPTO could implement to address patent quality and inclusive innovation, building on the agency’s existing commitment to continuous improvement, measurement, and excellence in administration. It is this project’s hope and conceit that academics and others will consider framing their policy recommendations, as pilots that agencies can try and learn from, not only changes to implement. As the DC Circuit has said, “in some situations, ‘a month of experience will be worth a year of hearings,’”133 and what agencies need isn’t necessarily to be told what the answer is, but how to discover workable answers on their own.

  1. [1]. U.S. Const. art. I, § 8, cl. 8.

  2. [2]. 35 U.S.C. § 2(a) (2012).

  3. [3]. Which stands for questions that matter, relevant authority, the underlying theory of change, testing protocol, evidence and resources.

  4. [4]. Collen V. Chien, Rigorous Policy Pilots: Experimentation in the Administration of the Law, 104Iowa L. Rev.2313, 2339–48 (2019) [hereinafter Chien, Rigorous Policy Pilots].

  5. [5]. For reasons discussed in the companion article, many academic articles are focused on other topics.

  6. [6]. U.S. Const. art. I, § 8, cl. 8.

  7. [7]. An invention must also cover statutory subject matter and be useful (35 U.S.C. § 101) and be adequately claimed and precisely disclosed. (35 U.S.C. § 112).

  8. [8]. Letter from Thomas Jefferson to Isaac McPherson (Aug. 13, 1813), available at [].

  9. [9]. P.J. Federico, Commentary, Commentary on the New Patent Act, 75. J. Pat. & Trademark Off. Soc’y 160, 164 (1993). Examination was restored by the Patent Act of 1836. Id.

  10. [10]. eBay Inc. v. MercExchange, L.L.C., 547 U.S. 388, 397 (2006) (Kennedy, J., concurring). Id. at 396 (describing the contribution of these patents to the development of “[a]n industry ... in which firms use patents not as a basis for producing and selling goods but, instead, primarily for obtaining licensing fees”).

  11. [11]. Including inter partes review, post-grant review (which covers the nine months after a patent has been granted), and covered business method review; for an overview of each, see Joe Matal, A Guide to the Legislative History of the America Invents Act: Part II of II, 21 Fed. Cir. B.J. 539, 623 (2012).

  12. [12]. U.S. Patent & Trademark Office, Patent Trial And Appeal Board Statistics 10 (2019), available at _february.pdf [] (indicating that 81 percent of all claims reviewed in inter partes review were found to be unpatentable).

  13. [13]. Rob Sterne & Gene Quinn, PTAB Death Squads: Are All Commercially Viable Patents Invalid?, IP Watchdog (Mar. 24, 2014), [].

  14. [14]. See infra Part III.

  15. [15]. At least certain kinds. During the first century of the patent system, slaves, non-white foreigners, and married women faced structural barriers to patenting. See infra notes 109–15 and accompanying text; see also Colleen Chien, Tracking Innovators (working paper on file with the author, at Part I); Olatunde C.A. Johnson, Overreach and Innovation in Equality Regulation, 66 Duke L.J. 1771, 1777 (2017) (describing “inclusion” as implicating barriers not just based on identity but poverty and geography).

  16. [16]. Peter Drahos, The Global Governance Of Knowledge: Patent Offices and Their Clients 99–109 (2010) (describing U.S. fees as set below U.K. fees at the outset, in 1790, and lower than most European countries, for the first half of the 19th century).

  17. [17]. Petra Moser, Patents and Innovation in Economic History, 8:241-258 Annual Rev. of Econ. 241, 247 (2016).

  18. [18]. Act of Aug. 27, 1982, Pub. L. No. 97-247, § 1, 96 Stat. 317, 317. For additional history about the introduction of reduced fees, see Jeff A. Ronspies, Comment, Does David Need a New Sling? Small Entities Face a Costly Barrier to Patent Protection, 4 J. Marshall Rev. Intell. Prop. L. 184, 193 (2004).

  19. [19]. Leahy-Smith America Invents Act, Pub. L. No. 112-29, § 10(b)–(g), 125 Stat. 284, 316–18 (2011) (establishing micro-entity fees and defining a “micro entity” as an inventor with fewer than four patents and whose income did not exceed three times the median household income for the preceding calendar year).

  20. [20]. Office of Chief Economist, U.S. Patent & Trademark Office, Progress and Potential: A Profile of Women Inventors on U.S. Patents 3 (2019), available at [].

  21. [21]. See infra notes 100–24 and accompanying text.

  22. [22]. The USPTO does not ask for demographic information from inventors, but it does ask for name data, from which a US origin inventor’s gender can be inferred at some level of reliability. However, the ethnicity of, in particular African-American and mixed-race, inventors is much harder to infer from names alone, including for historical reasons. Described in Chien, Tracking Innovators, supra note 15, at Part II.

  23. [23]. Alexander M. Bell et al., Who Becomes an Inventor in America? The Importance of Exposure to Innovation 34 (Nat’l Bureau of Econ. Research, Working Paper No. 24,602, 2019), [].

  24. [24]. Leahy-Smith America Invents Act, Pub. L. No. 112-29, § 29, 125 Stat. 284, 339 (2011). This provision was introduced in order to, in the words of its author Rep. Gwen Moore, overcome the gap in information that “prevents us from fully understanding the nature and scope of the underrepresentation of minority communities in intellectual property.” 157 Cong. Rec. H44,804 (daily ed. June 23, 2011) (statement of Rep. Moore). See U.S. States Patent & Trademark Office, Study and Report on the Implementation of the Leahy-Smith America Invents Act 48 (2015), available at Implementation_of_the_AIA_September2015.pdf [].

  25. [25]. Study of Underrepresented Classes Chasing Engineering and Science Success (SUCCESS) Act of 2018, Pub. L. No. 115-273, § 2(b), 132 Stat. 4158, 4158.

  26. [26]. Chien, Rigorous Policy Pilots, supra note 4.

  27. [27]. 35 U.S.C. § 2(A) (2012).

  28. [28]. U.S. Patent &Trademark Office, FY 2018 Performance and Accountability Report 56 (2018), available at [].

  29. [29]. 35 U.S.C. § 2(E).

  30. [30]. Chien, Rigorous Policy Pilots, supra note 4.

  31. [31]. The EPO which is generally recognized by applicants and other examiners as granting high quality patents. For cites to surveys and related literature, see generally Colleen Chien, Comparative Patent Quality, 50 Ariz. St. L.J. 71 (2018) [hereinafter Chien, Comparative Patent Quality] (surveying, inter alia, the quality differences amongst patent-granting organizations).

  32. [32]. For example, due to other differences in applicant behavior. One way to control for these differences is to perform a robustness check based on whether the patent was first filed in the US or abroad.

  33. [33]. See Colleen V. Chien et al., Inter Partes Review and the Design of Post-Grant Patent Reviews, 33 Berkeley Tech. L.J. 817 (2019) [hereinafter Chien et al., Inter Partes Review] (describing each review procedure).

  34. [34]. Pursuant to 37 C.F.R. § 1.56(a), the Duty of Disclosure provides that, “[e]ach individual associated with the filing and prosecution of a patent application has a duty of candor and good faith in dealing with the Office, which includes a duty to disclose to the Office all information known to that individual to be material to patentability as defined in this section.” 37 C.F.R. § 1.56(a) (2012).

  35. [35]. Precision Instrument Mfg. Co. v. Auto. Maint. Mach. Co., 324 U.S. 806, 819 (1945).

  36. [36]. For example, patent examiners, patent prosecutors, patent trial judges, or patent litigators.

  37. [37]. Jason D. Reinecke, Is the Supreme Court’s Patentable Subject Matter Test Overly Ambiguous? An Empirical Test, 2019 Utah L. Rev. 581, 583 (outlining a methodology comparing patent prosecution attorney predictions of district court rulings on § 101 and finding them to be accurate 67.3% percent of the time).

  38. [38]. Comm. on the Judiciary of the U.S. Senate, Report of the President’s Commission on the Patent System 13 (1966), available at [].

  39. [39]. Microsoft Corp. v. i4i Ltd. P’ship, 564 U.S. 91, 96 (2011).

  40. [40]. 35 U.S.C. § 282(a) (2012) (“A patent shall be presumed valid.”).

  41. [41]. Microsoft, 564 U.S. at 111 (emphasis added).

  42. [42]. Id. at 110.

  43. [43]. See, e.g., Chien, Comparative Patent Quality, supra note 31, at Figs 7A–7D.

  44. [44]. Such as, in the case of artificial intelligence. Artificial Intelligence, Cornell University:, [].

  45. [45]. See, for example, most recently, the Access to Prior Art Initiative of the USPTO. Access to Relevant Prior Art Initiative, USPTO, [].

  46. [46]. Letter from Alan J. Kasper, President, Am. Intellectual. Prop. Law Ass’n, to David J. Kappos, Director, U.S. Patent & Trademark Office 4 (June 23, 2010), available at http:// [https://].

  47. [47]. See Chien, Comparative Patent Quality, supra note 31, at 110–18. These techniques permit the distinct tracking of examiner and application references (on Form PTO-892 “Notices of References Cited and Form PTO-1449, Information Disclosure Statement (IDS) respectively). In the present and in previous studies, I rely on data reported on Form 892 to determine the extent of examiner reliance on non-patent literature. As the USPTO has previously acknowledged, there is a possibility that examiners exclude from the Form-892 references that they rely upon because they already show up in the 1449. To test the extent of this exclusion, I worked with a research assistant to manually hand-code 84 patents, finding a difference in outcome in 2 of them.

  48. [48]. See infra text accompanying note 90 (showing that 79 percent of office actions involve obviousness rejections based upon Author’s analysis of the USPTO Office Action dataset).

  49. [49]. Andrei Iancu, Director, U.S. Patent & Trademark Office, Role of U.S. Patent Policy in Domestic Innovation and Potential Impacts on Investment, Keynote Address at the U.S. Chamber of Commerce Patent Policy Conference (Apr. 11, 2018), available at about-us/news-updates/remarks-director-andrei-iancu-us-chamber-commerce-patent-policy-conference [].

  50. [50]. To do this I exploited a change in the way in which prior art references are reported by the USPTO. Though it has historically been difficult to distinguish examiner-relied upon references from others, that changed in 2001 when the USPTO started starring the results of examiner-searches on the front pages of patents. See Chien, Comparative Patent Quality, supra note 31, at 114. However, for the purposes of the analysis described below, I relied directly on the USPTO electronic database and considered prior art cited in the examiner’s search report (Form 892), as distinguishable from prior art cited in the information disclosure statement (Form 1449), provided by the patentee. There remains a possibility that an examiner will rely upon but fail to include in the 892 Form a reference originally sourced from a 1449 Form. To check this possibility, I worked with a research assistant to hand-inspect 84 file histories and compare the rate of examiner reliance on NPL as a result of hand-coding to the rate of NPL citation based on the 892 Form, finding a 2.5 percent difference (with citation rates 12 percent v. 9.5 percent, respectively), meaning that the examiner-NPL citation rates reported herein may be depressed but only by 15 percent. Recent work by Kuhn and others also finds that the public data reported by the USPTO may also attribute exclusively to examiners references that are also found by applicants, however it is unclear if this impacts the relative share of patent to non-patent literature. Further investigation is warranted. See Kuhn et al., Patent Citations Reexamined, RAND J. Econ.(forthcoming).

  51. [51]. Chien, Comparative Patent Quality, supra note 31, at 140 figs. A1 & A2.

  52. [52]. Id. at 140 fig. A2.

  53. [53]. I am thankful to Unified Patents for providing the list of inter partes reviews and their outcomes, which served as the basis for the analysis, and to Lex Machina, which enabled me to approximate citation patterns in inter partes review initial determination decisions.

  54. [54]. See infra Figure 1.

  55. [55]. When a single patent was subject to multiple inter partes review institution decisions, I picked the first one.

  56. [56]. See infra Figure 1.

  57. [57]. See infra Figure 2.

  58. [58]. Author’s calculations.

  59. [59]. Work is underway to explore the implications of a variety of aspects of prior art search and citation on patent quality.

  60. [60]. Out of 202 European twins, approximately 121 were granted, 64 were not granted, and 17 were pending.

  61. [61]. 52 percent vs. 20 percent.

  62. [62]. 23 percent vs. 13 percent.

  63. [63]. Admissions made by the patentee in the application or during prosecution, or so-called “applicant admitted prior art” can be relied upon to defeat the patent, including in IPR determinations. Riverwood Int’l Corp. v. R.A. Jones & Co., 324 F.3d 1346, 1354 (Fed. Cir. 2003); Constant v. Advanced Micro-Devices Inc., 848 F.2d 1560, 1570 (Fed. Cir. 1988).

  64. [64]. Request for Comments on Enhancing Patent Quality, 80 Fed. Reg. 6475, 6479 (Feb. 5, 2015); see also USPTO-led Executive Actions on High Tech Patent Issues, USPTO, http:// [https://].

  65. [65]. See generally Christopher A. Cotropia et al., Do Applicant Patent Citations Matter?, 42 Res. Pol’y 844 (2013) (finding that examiners were much more likely to cite the references they found rather than references provided through IDS). But see Kuhn et al., supra note 50 (finding, based on new data, the difference to be smaller than previously thought, but examiners still to rely on their own search results to a greater degree).

  66. [66]. Determining if examiner-citation of NPL was related to patent quality would be feasible with only a small sample of approximately 200 patents (N = 193 for 80% power at a correlation of 0.2) assuming a small positive relationship between amount of NPL cited and patent quality.

  67. [67]. eBay Inc. v. MercExchange, L.L.C., 547 U.S. 388, 396–97 (2006) (Kennedy, J., concurring).

  68. [68]. Mark A. Lemley & Kimberly A. Moore, Ending Abuse of Patent Continuations, 84 B.U. L. Rev., 63, 74–77 (2004).

  69. [69]. Iain M. Cockburn et al., Are All Patent Examiners Equal? Examiners, Patent Characteristics, and Litigation Outcomes, in Patents In The Knowledge-Based Economy 19, 28 (Wesley M. Cohen & Stephen A. Merrill eds., 2003).

  70. [70]. Alan C. Marco & Richard D. Miller, Patent Examination Quality and Litigation: Is There a Link?, 14–15 (USPTO Econ. Working Paper No. 2017-09, 2018), abstract=2995698.

  71. [71]. See Michael D. Frakes & Melissa F. Wasserman, Is the Time Allocated to Review Patent Applications Inducing Examiners to Grant Invalid Patents?: Evidence from Micro-Level Application Data 11 (Nat’l Bureau of Econ. Research, Working Paper No. 20,337, 2014), abstract=2467262.

  72. [72]. Brian J. Love et al., Determinants of Patent Quality: Evidence From Inter Partes Review Proceedings, 90 Colo. L. Rev. 67, 74 (2019).

  73. [73]. Marco & Miller, supra note 70, at 23.

  74. [74]. Brian Fung, Inside the Stressed-out, Time-crunched Patent Examiner Workforce, Wash. Post (July 31, 2014 9:18 AM), 07/31/inside-the-stressed-out-time-crunched-patent-examiner-workforce [ FM6Q-FWFR].

  75. [75]. U.S. Gov’t Accountability Office., Intellectual Property: Patent Office Should Define Quality, Reassess Incentives, and Improve Clarity 25 (2016), available at [].

  76. [76]. See, e.g., Frakes & Wasserman, supra note 71, at 9–11.

  77. [77]. See Press Release, USPTO, Recently Announced Changes to USPTO’s Examiner Count System Go into Effect (Feb. 18, 2010), available at [] (announcing “[m]ore time for examiners”).

  78. [78]. Mark A. Lemley, Fixing the Patent Office 18 (Nat’l Bureau of Econ. Research, Working Paper No. 18,081, 2012),; see also John R. Allison and Starling D. Hunter, On the Feasibility of Improving Patent Quality One Technology at a Time: The Case of Business Methods, 21 Berkeley Tech. L.J. 729, 734 (2006).

  79. [79]. Press Release, USPTO, USPTO Announces Action Plan Targeted to Improve Quality and Respond to Customer Needs for Biotech Patent Applications (Oct. 6, 2003), available at [].

  80. [80]. Chien, Comparative Patent Quality, supra note 31, at 99–110.

  81. [81]. See 35 U.S.C. § 41(h) (stating that fees for small entities are to be reduced by 50 percent); see also American Inventors Protection Act of 1999, Pub. L. No. 106-113, §§ 4711–4712, 113 Stat. 1501A–552, 1501A–573 (providing that the USPTO “shall recognize the public interest in continuing to safeguard broad access to the United States patent system through the reduced fee structure for small entities”).

  82. [82]. The fee change is described in New Fees and Micro Entity Status Take Effect March 19, Inventors Eye, [ 4BYM-89AY] (requiring that microentities must, among other requirements, “not be named on more than four previously filed applications” and “not have a gross income more than three times the median household income in the previous year”).

  83. [83]. Charles Dickens, Tale of a Patent 3 (1850) (describing an applicant so tortured by the patenting process that he asks, “[i]s it reasonable to make a man feel as if, in inventing an ingenious improvement meant to do good, he had done something wrong? How else can a man feel, when he is met by such difficulties at every turn? ... How hard on me [is the process of applying for a patent] to put me to all that expense ... .”).

  84. [84]. Colleen Chien, Inequality, Innovation, and Patents 35–43 (Santa Clara Univ. Legal Studies Research Paper No. 2018-03, 2018), =3157983. From 2005 to the present, the entity size missing data rate among USPTO applications is less than 2 percent of while from 2000–2005, the shares range from 6.5–10 percent. (Jan. 29, 2018 email from Rick Miller, USPTO, on file with the author).

  85. [85]. Id., at Figure 3B. Because patent fees are paid up to 10.5 years following issuance, and it takes, on average, 2–4 years to get a patent, this Essay reports on patents filed until 2004 to minimize truncation effects.

  86. [86]. Id.

  87. [87]. See 35 U.S.C. § 112 (2012) (“The specification shall contain a written description of theinvention, and of the manner andprocessof making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by theinventororjoint inventorof carrying out theinvention.“).

  88. [88]. Id. § 112(b) (requiring that a patent specification “conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his or her invention”).

  89. [89]. Id.

  90. [90]. Id.

  91. [91]. Jim Dwyer & Marty Rater, PowerPoint Presentation from USPTO, Examiners Provide Their Views on Prepared Applications: Application Readiness Survey, at slide 16 (Oct. 12, 2017), available at [ ] (reporting greatest and second greatest gaps, of 4.4 and 3.8, respectively, as to “claims that are solely directed to the inventive concept” and “independent claims that capture the same inventive concept disclosed in the specification”).

  92. [92]. Based on an analysis performed by the author and her research assistant of office actions in the USPTO OA Dataset from 2008–2017. Even though small and micro entities also experience higher levels of § 112(a) rejections than do large entities, the differences between small and micro entities, and small/micro and large entities § 101 rates are smaller.

  93. [93]. See, e.g., Colleen V. Chien, Powerpoint Presentation Can AI Fix Patent Quality?, at slide 30 (Mar. 18, 2018), available at [] (describing Turbopatent’s “RoboReview” product, which has trained an artificial intelligence engine to review patent application drafts and identify 112 clarity issues that stem from defects in the specification).

  94. [94]. Powerful Patent Tools that are Efficient and Secure, ClaimMaster: Patent Quality Assurance, [].

  95. [95]. See, e.g., National Science and Technology Council, Social and Behavioral Sciences Team: 2016 Annual Report 8–10 (2016), available at 2016%20SBST%20Annual%20Report.pdf [] (describing tools used by an agency to increase efficiency).

  96. [96]. Paul A. Showalter, Using 2-D Barcode Technology to Bridge the Paper-to-Digital Divide, Internal Revenue Serv.: Session 68, at 3, resmgr/articles/05_68.pdf [].

  97. [97]. 2-D Bar Coding for Schedules K-1 Is the Preferred Method, IRS (last updated May 31, 2019), [].

  98. [98]. Id.

  99. [99]. Showalter, supra note 96. The IRS’ experimentation with the adoption of barcodes could inform the design of a pilot with error-correcting patent quality software. The steps carried out by the IRS included: securing funding and buyoff to conduct the experiment, working with the software community to adapt existing technology to the IRS’ needs, identifying five tax preparation companies willing to pilot the technology, and educating the community and stakeholders on the objectives and merits of the idea.

  100. [100]. Adams Nager et al., Info. Tech. & Innovation Found., The Demographics of Innovation in the United States, 8–9 n.8 (2016), available at [].

  101. [101]. Jessica Milli et al., Inst. For Women’s Pol’y, Equity in Innovation: Women Inventors and Patents Res. 9 fig.5 (2016), available at wpallimport/files/iwpr-export/publications/C448%20Equity%20in%20Innovation.pdf [https://].

  102. [102]. Jennifer Hunt et al., Why Don’t Women Patent? 1 (Nat’l Bureau Econ. Research, Working Paper No. 17,888, 2012), (documenting a 9–10% female inventor rate); see also Office of Chief Economist, U.S. Patent & Trademark Office, supra note 20 at 3.

  103. [103]. See generally Lost Einsteins: Lack of Diversity in Patent Inventorship and the Impact on America’s Innovation Economy: Hearing before the Subcommittee on Courts, Intellectual Property, and the Internet, 116th Cong. 23 (2019) (statement of Dr. Lisa D. Cook, Mich. St. Univ.) (summarizing studies by Celik and others that try to estimate the loss in GDP due to this misallocation of talent, finding it to be roughly equivalent of three to four percent).

  104. [104]. See id. at 25.

  105. [105]. Corinne A. Moss-Racusin et al., Science Faculty’s Subtle Gender Biases Favor Male Students, 109 Procs. Nat’l. Acad. Sci U.S. 16,474, 16,474 (2012).

  106. [106]. Despite scant evidence of a talent gap in math, research by Stoet and Geary suggests that the “relative advantages” of girls and women in reading over math contribute to women selecting into non-STEM (rather than being excluded out of STEM) careers. See generally, Gijsbert Stoet & David C. Geary, The Gender-Equality Paradox in Science, Technology, Engineering, and Mathematics Education, 29 Psychol. Sci. 581 (2018) (explaining that gender does not affect performance in STEM areas); see also Olga Khazan, The More Gender Equality, the Fewer Women in STEM, Atlantic (Feb. 18, 2018), [].

  107. [107]. See Khazan, supra note 106.

  108. [108]. Office of Chief Economist, U.S. Patent & Trademark Office, supra note 20, at 3.

  109. [109]. Press Release, USPTO, First U.S. Patent Issued Today in 1790 (July 31, 2001), available at [https://].

  110. [110]. Office of Chief Economist, U.S. Patent & Trademark Office, supra note 20, at 3.

  111. [111]. See Fetter v. Newhall, 17 F. 841, 843 (C.C.S.D.N.Y. 1883) (confirming that minors, married women, and individuals suffering from a legal disability could apply for and own patents under the Patent Act).

  112. [112]. Kyle L. Jensen et al., Gender Differences in Obtaining and Maintaining Patent Rights, 36 Nature Biotechnology 307, 307–09 (2018).

  113. [113]. Id. at 307. Though reported only in the appendix, not main article, these differences were robust and in fact grew when the models included only US inventors (Table S13) and excluded pending applications (Table S14).

  114. [114]. Id.

  115. [115]. Moss-Racusin et al., supra note 105, at 16,475.

  116. [116]. Holly O. Witteman et al., Are Gender Gaps Due to Evaluations of the Applicant or the Science? A Natural Experiment at a National Funding Agency, 393 Lancet 531, 531 (2019).

  117. [117]. See, e.g., Diego F. M. Oliveira et al., Comparison of National Institutes of Health Grant Amounts to First-Time Male and Female Principal Investigators, 321 JAMA 898, 898–900 (2019) (finding that first-time women principal investigators received $41,000 less in funding than men on average, despite statistically indistinguishable records, based on number of articles and citations across a range of fields).

  118. [118]. Witteman et al., supra note 116, at 535.

  119. [119]. Milli et al., supra note 101, at 18–20.

  120. [120]. 7 percent–21 percent of 12 percent, based on Jensen et al., supra note 112.

  121. [121]. Jensen et al., supra note 112, at Online Appendix, Table S14.

  122. [122]. In a similar vein, an applicant’s decision to withdraw from prosecution before the patent has begun substantive examination, or “early abandonment” reflects only the applicant’s, not the examiner’s behavior, and could be used to isolate the impact of a persistence effect on differences in grants.

  123. [123]. Jensen et al., supra note 112, at 309.

  124. [124]. For a preliminary analysis showing this is a case, see Colleen V. Chien, Senate Judiciary Committee’s Intellectual Property Subcommittee Hearing on Patent Quality on Oct. 30, 2019 (available at

  125. [125]. To observe an effect size of 20 percent would require as few as 93 observations per control and treated group, and an effect size of 7 percent would require more like 771 observations per group.

  126. [126]. My analysis suggests that out of 3.5M unique inventor names in the PatentsView database, 537K are missing first names, and 760k have only one to two letter names, suggesting that this strategy could be exploited successfully.

  127. [127]. See 37 C.F.R. § 1.76 (2018).

  128. [128]. See id. § 1.63.

  129. [129]. Letter from USPTO to author (May 29, 2019) (on file with the author).

  130. [130]. 37 C.F.R. § 1.48(f).

  131. [131]. Id. § 1.17(i).

  132. [132]. Which stands for questions that matter, relevant authority, the underlying theory of change, testing protocol, evidence and resources. Chien, Rigorous Policy Pilots, supra note 4.

  133. [133]. Maryland People’s Counsel v. FERC, 761 F.2d 768, 778–79 (D.C. Cir. 1985).


Justin D’Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013–2015 White House Senior Advisor, Innovation and Intellectual Property. 

Professor of Law, Santa Clara University, Fellow, Stanford Computational Policy Lab, 2013–2015 White House Senior Advisor, Intellectual Property and Innovation. I thank United Patents, Lex Machina, and Google Patents for making public patent data accessible for this project, Nick Halkowski for excellent research assistance, and the student editors of the Iowa Law Review for their hard work and dedication to this project. This is a companion Essay to Rigorous Policy Pilots: Experimentation in the Administration of the Law, 104 Iowa L. Rev. 2313 (2019). Substantial portions of this Essay served as the basis for testimony before the Senate Judiciary Committee’s Intellectual Property Subcommittee Hearing on Patent Quality on Oct. 30, 2019 (available at and during SUCCESS Act Hearings by the USPTO on June 4, 2019 (