Software problems caused Healthcare.gov glitches, U.S. official says
Poorly written code, out-of-date applications found in outside analysis
IT contractor CGI-Federal had received $90 million by March for site, GAO says
Government won't say what parts of the site CGI-Federal built
Technical problems with the federally facilitated health insurance exchange were caused by a specific software component that crashed under the high volume of visitors last week, preventing users from creating accounts in the beginning stages of the enrollment process, a federal official confirmed to CNN on Tuesday.
“At lower volumes, this software component would work fine, and at higher volume, that’s what created the problems,” the official said. “We’ve made software changes to make the system more efficient and be able to handle higher volumes.”
For more than a week, visitors to Healthcare.gov have experienced slow connection speeds and error messages preventing them from purchasing insurance. The Department of Health and Human Services has been taking down parts of the site during off-peak hours, and officials say that adding more server capacity; moving certain over-stressed components to dedicated hardware; and making software changes to increase efficiency have improved the situation.
On Monday, HHS Secretary Kathleen Sebelius told Comedy Central’s Jon Stewart that despite the hiccups, hundreds of thousands of consumers have created accounts on the federal site, but the government won’t announce how many of those users have actually purchased insurance until next month.
The government has been assisted in building and maintaining Healthcare.gov primarily by contractor CGI-Federal. As of March, the contractor had collected nearly $90 million from the Centers for Medicare & Medicaid Services, according to a report issued in June by the Government Accountability Office.
“We have a unique, cross-sector perspective on health IT that makes us well-suited to partner with CMS for success on one of the most visible efforts in CMS history,” CGI Vice President Rich Martin said in a December 2011 statement announcing the contract, the original value of which was more than $55 million.
As part of Obamacare implementation, CGI is also performing exchange work in California, Colorado, Hawaii, Illinois, Massachusetts, Kentucky, New Mexico and Vermont, according to the company’s website.
A Canadian company with a global presence, CGI presents itself as the world’s fifth largest independent IT and business process services company. The company has experience in health care IT as a partner to some of the Canadian Ministries of Health and as a provider of secure electronic medical records in Europe.
Its U.S. government contracting branch, CGI-Federal, has done more than $2.5 billion in business with more than two dozen government agencies since 2001, according to a search at USASpending.gov. This work includes nearly a billion dollars of work with HHS, costly projects for the defense and intelligence industry, as well as contracts worth potentially billions of dollars supporting the U.S. Army, Navy and Marine Corps.
Career operations staff at both HHS and CMS managed the company’s work, as well as the work of Quality Software Services, Inc., a separate contractor that has received about $55 million from CMS to build the federal data hub.
According to government officials, that data hub and the various levels of identity verification necessary for enrollment are working properly, providing accurate eligibility results and tax credit determinations. The problems have been centered in the website’s front-end, or Healthcare.gov’s consumer-facing side.
An analysis of this front-end performed by leading web-hosting company Media Temple for the Wall Street Journal and provided to CNN found that the designers failed to follow basic protocols for high-traffic sites. Six of Media Temple’s high-level engineers point to the site’s structure rather than web traffic as the primary problem. The faults they identified include out-of-date web applications; poorly written code; lines of testing code left in the final product; and unnecessary or unchanging information sent to users on each login rather than stored on local servers or computers for quicker access, a process called caching.
“We’re in the business of fast websites that don’t fail, that’s what we do,” said Media Temple President and COO Russ Reeder. “And so when I had my engineers look at the web site, they came up with a lot of things that they could have done better. If this is so important and they’ve gotten paid so much money – the consulting firm CGI, $55 million for this project where’s the oversight? Where’s the technical leadership saying, ‘tell me how you tested this and how do we make sure that this is not only secure but it won’t fail when we have 7 million people log in?’ “
Government officials wouldn’t comment on which parts of Healthcare.gov were built by CGI and which parts were built by the government. CGI didn’t return requests for comment.