Balancing Fairness, Accuracy, and Explainability in Machine Learning Models for Credit Decisioning
By Graydon Goss As machine learning (ML) reshapes the landscape of credit decisions, the stakes around lending fairness become much more than an…
Promoting Equity Through Procurement Requires Rethinking How We Use Data
BY TOM COLEMAN and BRIDGETTE SULLIVAN
President Biden’s first Executive Order upon taking office sought to leverage the federal government’s existing powers to advance racial equity.[1] Expanding opportunities for historically marginalized entrepreneurs and their communities to participate in government procurement was expressly cited by the Executive Order as a means of accomplishing this goal. However, without effective ways of measuring opportunities afforded to minority-, women-owned, and other disadvantaged small businesses, agencies will not know how to achieve the Executive Order’s goal. This is because the existing practice for measuring small business opportunities is over-simplified, outdated, and arguably arbitrary. The solution is for agencies to use innovative data capture and analytics practices to meet and exceed the goals of the Executive Order by modernizing how it uses its data to both level the playing field and create equity and opportunity for historically marginalized entrepreneurs.
The Current Measurement System Encourages Arbitrary Goal Setting
Earlier this year, I attended a thoughtful and engaging small business forum for a cabinet-level agency. During that event, a senior procurement official was asked by an audience member why their agency recently lowered its annual small business contracting goal. The response was disappointing, but refreshingly honest – The agency had missed their small business goals the past few years and therefore a lower target was needed to claim future success. This candid remark underscores a fundamental challenge in how the government measures small business contracting.
While agencies must set annual small business goals, including goals for minority-owned small businesses, there is little basis for deriving specific percentage goals, beyond meeting statutory minimums and striving for prior year improvement.[2] Meeting a single-metric percentage-of-spend goal is often a balancing act between demonstrating improvement over prior years and setting a goal that is deemed obtainable. The problem with this single-metric assessment is that it uses the lone benchmark of prior year performance while failing to take advantage of the wealth of available procurement data to question whether the goal is reasonable or if achieving the goal creates new opportunity. In essence, the current system incentivizes agencies to “perform a measure correctly” at the expense of “measuring the correct performance.”
What Do Percentages Tell Us About Opportunity?
Measuring access to opportunity requires going beyond the ineffective percentage-of-spend metric. A recent GAO report noted that while contracting spend increased more than 22 percent from 2016 to 2019, the number of small businesses receiving at least one government contract fell by 17 percent, or roughly 16,000 businesses during the same timeframe.[3] GAO stated the government’s concerted effort to consolidate contracts through Category Management and “Best-In-Class” vehicles limits opportunity and funnels contract dollars to a handful of the largest “small businesses.” The U.S. Small Business Administration (SBA) recognized that reversing the hemorrhaging of small businesses from the federal marketplace is one of the best options to create opportunities for new businesses and help close the equity gap.[4]
Even if percentage goals captured the optimal amount of spending that should be reserved for small and disadvantaged businesses, it fails to measure the expansion of opportunity. The contraction of the federal small business community despite rising small business spend illustrates the failure of percentage goals. For one, percentage goals do not measure how much small business spending ends up in the hands of large businesses through joint ventures or subcontracting. Further, percentage goals ignore the barriers to entry that block disadvantaged small businesses from entering the marketplace in the first place. Emerging small businesses compete on an uneven playing field against well-established contractors with up to $30 million in annual revenue, who are still considered small businesses.
The uneven playing field is best observed through the increased funneling of small business spend through Category Management initiatives including “Best-in-Class” procurement vehicles that require “small businesses” to hold expensive certifications and possess years of winning multi-million-dollar, prime contracts to even participate in business opportunities. New entrants, like those businesses the Executive Order intends to assist, often are denied the opportunity to even compete for, let alone win government contracts.
The single-metric method for analyzing small business opportunity is antiquated, fails to adequately leverage data, and reinforces structural barriers that prevent small and disadvantaged companies from participating in the federal marketplace. These negative outcomes are not borne from ill-intent. Most agencies put time and effort into meeting small business goals but without reimagining how we define and measure success through better data analytics, agencies cannot make data-driven policy decisions that increase equity in the government marketplace. Data analytics that look at new agency contractors, geographic distribution of vendors, size of small business contracts, gradients of contractor size, and subcontracting trends can enable agencies to expand opportunity for small and disadvantaged businesses and provide real-time assessment of the impacts of these important policies.
Why Agencies Need to Get Ahead of the Curve
The public has an increasing expectation that the Government demonstrate the value and impact of the significant taxpayer investments. As data-driven decision making becomes the norm in government, how agencies use data to achieve policy goals will garner increasing scrutiny. With agencies like OMB and NASA already taking steps to assess how the government combats equity in procurement, it is likely that, over the next year, early adopter agencies will show how they use data to assess and address equity gaps in procurement. Those not leading will be expected to follow, which is why agencies should be thinking expansively about their data capabilities now.
Pairing data professionals with technology that enables data visualization, counterparty exchanges, and sound data management will be critical to meet the policy mandates of the next few years and beyond. Advanced data analytics can vastly exceed the utility of the decades-old and flawed mechanism for measuring small business opportunity by percentage-of-spend alone. Agencies that apply enhanced data analytics to their procurement portfolios, will be better positioned to make tactical decisions that promote equity, demonstrate their successes to the broader acquisition community, and position themselves as a government innovator that will help set the standards of tomorrow.
Agencies Can Achieve Better Data Analytics
While the lack of analytics related to small business contracting is a government-wide problem, individual agencies can use available data to make an immediate impact on how to promote emerging small business participation. Existing procurement data – widely available to the public at USASpending.com – can be paired with other data sets (i.e. Bureau of Labor Statistics, internal agency data, SAM.gov) to uncover an agency’s concentration of small business spend, diversification of small businesses, incumbency advantage, and year-over-year government and industry trends to better understand how they engage small businesses and identify areas for improvement. Such measures do not require new legislation or policy direction, as seen by agencies like the IRS have already pursued advanced procurement data analytics for other purposes.[5]
Advances in technology present a host of options for ingesting and analyzing data that did not exist several years ago. The proliferation of data skills and tools enable agencies to do more with data without feeling budget pain. Highly flexible and affordable commercial solutions can integrate seamlessly with existing agency assets to combine and analyze complex datasets. Technologies such as machine learning and robotic process information can create new ways of extracting and analyzing large, unstructured data sets to uncover previously unavailable trends and conclusions, and business intelligence (BI) packages exist in nearly every agency so data can be conveniently accessed and understood by critical decision-makers.
What Better Analytics Tells Us About Small Business Opportunity
To illustrate the value of advanced analytics, we analyzed the publicly available 2021 fiscal year data for a large cabinet-level agency. In the spirit of the Biden Executive Order, we used the data to assess how the sample agency engaged non-incumbent small businesses. (Figure 1 Below).
Using the traditional percentage of agency spend metric, this agency committed roughly 34% of its FY2021 dollars to small business – a figure largely in-line with agency goals. We went further, measuring the degree of access new, non-incumbent small businesses had to this agency.
When isolating FY21 contracts for companies without a contract in the prior five fiscal years, our data shows that only 4.6 percent of the agency’s FY21 spend went to non-incumbent companies. Moreover, a meager 0.7% was awarded to non-incumbent small businesses. In absolute terms, this uncovers the limited opportunity for emerging small businesses at this agency.
Only 15.6% of “new contractor” spend went to small businesses compared to 34% for small business spending across all contracts. This drastic drop-off in small business opportunity for new entrants shines a rare light on largely unexamined entry barriers that exist in the status quo.
Good Things Happen When Agencies Embrace Data Analytics
When individual agencies take a forward-leaning approach to data analysis, positive procurement outcomes can be achieved. One relevant case study is the U.S. Air Force’s (USAF) approach to Category Management. As noted earlier, the government’s general approach to Category Management creates barriers to small businesses entry by funneling increasing amounts of government spending through exclusive contracts awarded to established contractors. USAF took a different, data-driven approach to Category Management that achieved verifiable savings without sacrificing small business inclusion.
USAF’s emphasis on data-driven decision-making involved:
- Ingesting information from a wide variety of federal and non-federal sources;
- Employing skilled data professionals and industry domain experts;
- Developing market intelligence reports for individual categories of goods and services; and
- Creating spending analysis tools for broad agency usage
Together, these measures have helped USAF better understand their spend to effectively manage demand and reduce costs.[6] USAF’s efforts resulted in more than $1 billion in savings over its first three years. Unlike broader government practices of Category Management which, according to GAO, focus more on the process on how to buy something as opposed to creating savings, USAF independently found a better way forward, saving money without excluding market participants. Said differently, shrinking your supply-base and reducing competition doesn’t save money, thoughtful analysis does. When agencies think differently about small business spend, they can promote broad participation, create opportunity and save money.
How To Get Started
With competing priorities for limited resources, the thought of embarking on an effort to strengthen data analytics capabilities can be daunting for procurement offices. Afterall, data analysis is a broad discipline, and it can be difficult to determine where to begin. But progress needs not be scary, and can be achieved incrementally through any one or a combination of the steps below:
- Creating Multifaceted Small Business Measures. The previously cited GAO study observed that solely measuring percentage spend obscures whether the government provides opportunities to new and disadvantaged small businesses. However, combining percentage spend with other measures, such as concentration, average value, and number of an agency’s new small business contractors, can create a more robust picture of the opportunities for small and disadvantaged businesses within an agency and trendlines can uncover where barriers may exist at the agency-level or pinpoint challenges in specific spend categories or offices. Agencies are already required to produce and report the data necessary to perform this analysis and need only establish the data capabilities and metrics to greatly enhance the way they understand how their procurement function encourages or discourages equity.
- Pursuing Data Visualization and Business Intelligence (BI) Capabilities. The ability to visualize data in an intuitive manner is key for sound, data-driven policymaking. Government is increasingly taking advantage of BI tools to create public-facing dashboards to address areas of public interest, administrative spending, and compliance. A range of easy-to-implement BI tools are available in the marketplace specifically to meet this need. Not only do BI dashboards enable public accountability, but internal dashboards can provide granularity to analyze spend by market segment or office to get a granular view of their metrics. Moreover, procurement offices and offices of minority and women inclusion can use BI dashboards to observe when agencies are at risk of falling behind on small business goals and take remedial actions to identify offices that are underperforming on goals and intelligent craft corrective actions.
Figure 2 demonstrates how business intelligence can use large data sets to tell a story. The graphic displays how subcontract plans were utilized in contracts to Large Businesses exceeding $750,000.[7] The visualization demonstrates that 67.4% of these contracts do not report having a small business subcontracting plan, with 20.1% of actions having a null value implying either a data error or noncompliance with subcontracting requirements. Moreover, 36.2% of actions note that there are no subcontracting possibilities. In my experience, this categorization is sometimes used to mask noncompliance and a higher-than-expected concentration of these designations within an office might be grounds for closer manual reviews. Data visualizations, like the one in Figure 2, can synthesize large amounts of data into digestible stories and help track and analyze critical metrics that point to the amount of opportunities that exist within an agency for emerging small businesses.
- Collection of Better Data from Industry: Some of the most important insights are not readily available in existing data sets. New technologies can enable agencies to capture data from contractors, bidders, or other external parties. This data can include qualitative impressions from emerging and historically disadvantaged businesses on their experience engaging with your agency. Combining quantitative data analytics with detailed anecdotal feedback can provide a holistic view of the experience of small and disadvantaged businesses. Many agencies already have tools and technology needed to support external data capture for other use cases, and need only to extend them to their procurement function to securely obtain additional industry data.
Conclusion
Bibi Hidalgo, SBA’s Associate Administrator for Government Contracting and Business Development, recently summarized the importance of eliminating barriers for disadvantaged contractors as follows: “This [federal procurement] is a huge — probably the largest avenue for wealth creation in the United States and potentially around the globe relative to other procurement budgets — and we have such a huge opportunity here to make sure that we create access to this opportunity.”[8] Making good on this opportunity requires a better understanding of the problem and a mechanism for measuring the impact of data-driven policies to expand opportunities. A solution is well within reach. By leveraging the tools, skills, and technology needed to capture and analyze data, agencies can vastly improve how they measure small business opportunity and address racial equity by providing increased access to many more driven, qualified, and diverse small business entrepreneurs.
References
[1] “Executive Order 13985 of January 20, 2021, Advancing Racial Equity and Support for Underserved Communities Through the Federal Government,” The White House. January 20, 2021, Retrieved June 29, 2021 https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/executive-order-advancing-racial-equity-and-support-for-underserved-communities-through-the-federal-government/ [2] The. U.S. Small Business Administration, FY 2021 Goaling Guidelines, (October 2020), Retrieved July 5, 2021 https://www.sba.gov/sites/default/files/2021-04/FY21%20Small%20Business%20Goaling%20Guidelines_Draft_202012Final-508.pdf [3] United States Government Accountability Office (GAO), OMB Can Further Advance Category Management Initiative by Focusing on Requirements, Data, and Training , GAO-21-40 (November 2020) https://www.gao.gov/assets/gao-21-40.pdf. [4] Ewall-Wice, Sarah “As the Biden Administration Pushes Major Spending, Small Businesses Fear “Missing Out.” cbsnews.com, June 11, 2021, https://www.cbsnews.com/news/biden-spending-small-businesses/. [5] See, for example, Boyd, Aaron “IRS Offering Up to $7M to Develop Data Mashing Tools in Support of Frictionlkess Acquisition.” Nextgov, September 4, 2020, https://www.nextgov.com/cio-briefing/2020/09/irs-offering-7m-develop-data-mashing-tools-support-frictionless-acquisition/168252/. [6] Cooke, Tim and Anne Laurent, “Air Force Adds Attitude to Governmentwide Category Management.” Federal News Network, October 14, 2019, https://federalnewsnetwork.com/commentary/2019/10/air-force-adds-attitude-to-governmentwide-category-management/ [7] The data used in the figure and the associated analysis is one month of 2021 data pulled from USASpending.gov for a cabinet-level agency and its subcomponents. [8] Ewall-Wice, “Small Businesses Missing Out,” June 21, 2021.
