When Mike Roberts applied for the XPRIZE Rapid Reskilling competition, he thought his application would be “a slam dunk.”
The competition, run by the Silicon Valley-based XPRIZE Foundation, would award grants totalling $5 million to programs that aimed to “quickly reskill under-resourced workers for the digital revolution.” Robert’s apprenticeship program, which paid participants to attend software engineering training and placed them in full-time roles, seemed like a great fit.
But when Roberts reached out to the XPRIZE team with questions about the application, it became apparent that his program wasn’t eligible. The reason: his apprenticeship was to train programmers, and according to data from the Bureau of Labor Statistics (BLS), programming was a field where the amount of jobs was projected to decline by 7.2 percent.
As education costs have risen, colleges and universities have been pressured to demonstrate that their programs produce measurable outcomes. In particular, many institutions have been criticized for enrolling more students in degree programs than there are jobs upon graduation. If colleges were more receptive to market data, the thinking goes, they would do a better job of steering students to programs that lead to available jobs.
Software development is one of the quintessential examples people point to, with various estimates putting the number of unfilled jobs at around one million. Yet a program designed to help fill that need was turned down precisely because the data said it wasn’t needed.
The big idea: Measuring which fields are in demand is harder than it sounds. Many of the available data sources, experts say, have significant flaws. And that causes problems for education providers who are trying to understand market demand and map their programs to it.
“If you are in higher education and trying to understand where the labor market is going, use BLS data as a general guide but do not rely too heavily on it when it comes to building programs and making investments,” said Jason Tyszko, the Vice President of the Center for Education and Workforce at the US Chamber of Commerce Foundation.
What’s In-Demand?
Why it matters: Colleges are turning to labor market data as they face increasing pressure from lawmakers and the public to demonstrate value and financial ROI. A number of states also have launched specialized grant and “free college” programs for residents pursuing education in high-demand fields. And many require state agencies to determine which fields are in high demand as part of workforce planning processes.
Virginia is one of those states. To comply with state law, the Board of Workforce Development has to regularly update a list of high demand occupations. Deciding how to do so can be challenging.
According to a presentation given at a September 2021 meeting, the board chose to determine which occupations are in high demand by using BLS data. The reason: the BLS data is publicly available.
“Although in some instances, proprietary data sources have different or additional nuances, in service of guiding principle #1 (transparency, replicability), our team has relied exclusively on publicly available data for this exercise,” the presentation said. (A representative from the board declined to comment, citing the still ongoing nature of constructing the high demand occupations list.)
The limits of the gold standard
For institutions looking to study job market trends, there are typically two main data sources available. The first, from BLS, are official government statistics primarily designed to track economic indicators such as the unemployment rate. The second, from proprietary companies such as Emsi Burning Glass, typically relies on postings to job board websites like LinkedIn.
The details: The two sources have different strengths and weaknesses. The Emsi Burning Glass data can be considered “real time” data, because it identifies new job postings as they are released online. The BLS data, on the other hand, is updated less frequently but is comprehensive.
The BLS data is designed to compare economic trends across decades, and to map to state systems so that statistics like unemployment rates can be compared across states. For those reasons, the agency is reluctant to change the definitions underlying the data. That consistency, however, can make it difficult for education providers to use the data to determine which fields are in high demand.
BLS data is broken down according to the Standard Occupation Classification system, or SOC, a taxonomy used to classify different occupations. That taxonomy is designed to be public facing—the BLS website, for example, features a guide for job seekers that purports to tell them which occupation codes have the highest wages or the greatest potential for growth.
But the taxonomy was last updated in 2010, according to a BLS spokesperson.
By erring on the side of consistency, experts say, the BLS’ data may reflect dated assumptions about how the U.S. labor market is structured. In many cases, those assumptions may not have been accurate to begin with.
The federal taxonomy represents “how government believes the labor market is structured,” Tyszko said.
“However, this taxonomy does not always reflect actual job titles and business functions. It is an attempt to define and structure the labor market, which is actually unstructured and incredibly heterogeneous.”
On the ground: That can trip up practitioners like Roberts. His program was categorized under “computer programmer,” which BLS defined as typically requiring a bachelor’s degree, per XPRIZE emails reviewed by Work Shift. At the time, BLS also projected that the field would decline by 7.2 percent. (The new 2020-2030 projections now have it declining by 10 percent). According to these numbers, any attempts to train programmers would likely not be fruitful.
But at the same time, a similar field, “software developers, quality assurance analysts, and testers,” is projected to grow by 22 percent. And while both fields typically require a bachelor’s degree, according to the BLS website, companies at least say that’s changing. Apple, Google, IBM, and Microsoft have all dropped degree requirements for entry-level technical roles, and are investing heavily in creating alternative pathways to tech jobs. The data don’t capture any of that—but they’re the gold standard that organizations like XPRIZE use.
“They were relying on a data source, and exclusively relying on that data source, and not putting the effort in,” Roberts said of his experience with the prize. “It would have taken minimal effort to determine where is industry at, where is the gap, where is there capacity for us to create well-paying jobs.”
An XPRIZE spokesperson said that the program used BLS data to filter out occupations with growth rates of less than 3.7 percent, and that it was focused on speed, cost, potential to scale, and the ability to “place graduates of their programs into select jobs.”
Other players in the market
The alternatives: Real time data from companies like Emsi Burning Glass and Revelio Labs promises to ameliorate the issues with BLS data by gathering job postings and other public data from the web in real-time. The nonprofit SkillsEngine, developed by the Center for Employability Outcomes at Texas State Technical College, similarly uses resume and job postings data to identify in-demand skills.
Emsi Burning Glass, the dominant company in the space, offers a skills taxonomy that is updated every two weeks, and also breaks its data down according to the SOC system.
“Our research, based on what employers are requesting in job postings, has shown that the specialized skills underlying the average job have changed up to 30 percent over the past 10 years,” said Isaac Lopez, an Emsi Burning Glass spokesperson.
But Emsi Burning Glass’ data also has limitations. Real-time labor market data is generated by crawling the internet and pulling together the available job posting data. That “can get the numbers wrong,” Tyszko said. Sometimes, for example, a single posting is meant to fill multiple job openings. Other times, a company is building a potential talent pool in case a new contract is secured, and those postings don’t reflect current openings.
Emsi Burning Glass defended the quality and scope of data, noting that the company’s research has shown the data represents 95 percent of the labor market. “There are some sectors, industries, and occupations that are underrepresented and we have been very transparent about it,” Lopez said. “For example, jobs filled through union hiring halls, or jobs in small businesses and restaurants, may not be posted online. However, jobs have been steadily shifting online.”
States join in: A number of regions and states have been working to combine real-time jobs data with information on education providers’ program-level outcomes and wage data from state unemployment insurance systems. The idea is to use multiple data sources to provide a more robust picture of what’s going on in the labor market—and the education and training options that lead to family-sustaining jobs.
The Data for the American Dream initiative is working with Colorado, Michigan, and New Jersey to bring multiple public and private data sources together and make them more transparent. The primary goal is to provide better information to education and jobseekers.
Those states are still refining their approach, but thus far, the impact of such efforts nationwide has been mixed, according to a recent report by the group. That’s due in part to data limitations. The unemployment insurance data in most states, for example, lacks information about specific occupation and only maps to a broad industry.
“The data infrastructure, while impressive, still has notable gaps and some of the important linkages are still in development,” wrote Patrick Lane, report author and vice president for policy analysis and research at the Western Interstate Commission for Higher Education.
Tapping employers directly
Going deeper: Colleges have a third option for measuring employer demand for specific occupations: talking to the employers themselves. That’s a method Tyszko advocates for through the Chamber of Commerce Foundation’s Talent Pipeline Management system. The system allows employers to organize and send detailed data on their hiring needs to colleges and other workforce partners.
“There is no substitute for getting the data directly from the employers,” said Tyszko. “Obtaining primary source data directly from the employers themselves will ensure you have actionable information about their jobs, including how they describe them, how many positions will be available, and the skills needed to be qualified for them.”
Employers like the program because it helps them work collectively to build trust with colleges, who can be reluctant to base funding decisions on the sometimes fickle needs of a single business.
“The very first colleges that we went to, it took a little bit of time to build the trust, because we had a misstep,” said Sharon Miller, the director of strategic talent pipelines at Consumers Energy. “We tried to put a program in place 10 years prior. Several schools were excited, and then we had a business change and didn’t hire the graduates.”
But now, with the Talent Pipeline Management system, Miller can collaborate with other employers and can create workforce plans for entire industries, which reduces the risk that one company might change its plans. “
“They are willing to start up a program when companies walk in [together] and say here’s our demand,” Miller said.
The human element
But even working with multiple employers has its limits. Reliable data is important, but how colleges use it is equally important. Institutions have to look out for the long-term careers of their students, while employers might only be thinking a few years ahead and discount concerns about automation or low wages.
“Labor market data can help colleges plan for the long-term success of their graduates beyond an individual employer’s immediate concerns,” said Shalin Jyotishi, a senior analyst at New America.
Consumer advocates also say that states should be cautious about allotting aid based on the field a student chooses to study. Women and people of color are disproportionately in low-paying fields, like education and social work, that have a high value to society but often don’t make it onto “high-demand” job lists. (Early childhood education is a notable exception right now.)
“Basing the cost of an education on students’ areas of study only further cements existing structural inequalities for women, people of color, and other historically marginalized populations,” said Ben Kaufman, the head of investigations at the Student Borrower Protection Center. “The higher education institution you attend and major you select both correlate strongly with race, gender, and other protected traits.”
In other words, advocates say, it’s not just enough to get better data—students and providers should be cautious about how they use it. Data for the American Dream also argues that data has to be paired with investment in guidance and counseling. “This final component of a well-aligned system of career opportunities and education and training provision may be the most difficult,” the group wrote.
Parting thought: In the meantime, practitioners like Roberts are left feeling frustrated. “I would love to have better data,” he said. “It would make working with funders easier.”
