0. Summary
Penetration testing is a critical
tool in ensuring the security of everything from an individual software
application to an entire network. Unfortunately, security is far too complex to
provide any sense of absolutes. Add to that the fact that dozens (if not
hundreds) of new vulnerabilities are discovered every week, and the need to
continuously test the security of a system is obvious.
We have provided services,
similar to those we provided HCFA, to hundreds of companies, and are
intricately aware of the “industry standard” state of security. During our
tenure providing security services to HCFA, we found both extremely positive
and disturbing issues. Major recommendations include:
Technical Oversight:
HCFA is lacking the
specially trained personnel to oversee their and their contractors’ activities
and verify the work for security consistent with policy and best practices..
Third-Party Verification:
It should be unacceptable
for service providers to certify themselves as secure. Any vendor of network services
to HCFA should readily accept 3rd party verification of security and
have regular testing a part of their contract performance requirements.
Security Specified in Contracts:
The security expectations
and requirements should explicitly be laid out in contracts with network
service providers.
More Testing is Required:
It’s necessary to
independently verify the security features of everything from applications to
WWW servers to networks and to do so on a recurring basis.
1. Background
En Garde Systems (EGS) provided a variety of
security services to the Health Care Financing Administration (HCFA) between
December 1997 and June 2000. During that time, EGS performed a number of
penetration tests and assisted HCFA in devising network security protections.
Specifically, EGS has performed:
-
External Penetration Tests (4). As an average outsider connected to the Internet,
we attempted to gain access to internal HCFA resources.
-
Internal Penetration Tests (2). Given a connection to HCFA’s internal network, we
attempted to gain access to internal HCFA resources.
-
Wardialing. Given a prototype phone number (for example 786-xxxx), we dial every
phone number looking for computer modems. When a computer answers, we attempt
to gain access.
-
Architecture Review and Design. Given a complete map of network resources, we
spent extensive time understanding the various applications HCFA provides and
the security needs of each. We then formulated network architecture changes
that would build security into the fabric of the network.
-
Test of Internet services hosted by IBM Global Services. At the time we tested,
HCFA outsourced its internal and external web servers to IBM.
-
NT Desktop Review. For Y2K, HCFA moved to a Windows NT desktop system. We were provided
with a prototypical NT desktop prior to deployment to find security
vulnerabilities.
-
HCFA Insider test. We were given a standard user’s desktop computer and asked to gain
access to HCFA internal resources. In this case, we were not allowed to bring
in our own software, floppies, or PC—only what we could retrieve using HCFA’s
network.
-
Intrusion Detection System Review. HCFA ran a “bake-off” of several competing
Intrusion Detection Systems and asked us to perform a variety of tests to
determine their efficacy.
-
Security Training. We provided classes over a several day period covering everything
from good password choice to Intrusion Investigation and Response.
2. Approach
Penetration testing is a critical tool in ensuring
the security of everything from an individual software application to an entire
network. Unfortunately, security is far too complex to provide any sense of
absolutes. Turning on one network service may result in dozens being turned in
non-obvious ways. Connecting a trusted partner to your network often means you
not only trust him, but everyone he trusts as well. Add to that the fact that
dozens (if not hundreds) of new vulnerabilities are discovered every week, and
the need to continuously test the security of a system is obvious.
Our approach to testing is almost exclusively
“manual”. We rarely use automated tools, as our experience has shown they are
generally only effective in an extremely small number of cases. Instead, we
learn about the network and create new attacks on the fly. In doing so, we are
doing exactly what a hacker does.
Proposing solutions to vulnerabilities is perhaps
the most complex part of our work. Before we recommend any solution, we need to
determine the:
1)
Value
of the organization’s data. If the data is simply pricing and personnel information,
it is far less valuable to a hacker than Privacy Act data, for example.
2)
Threat
to the organization. A government agency will attract far more interest from
the malevolent than a small computer company, for example.
3)
Path
of least resistance. It makes no sense to spend a great deal of effort
protecting a network connection to a partner if the “front door” is wide open.
These relative threats are determined before any solution is recommended.
4)
Cost in man-hours and
equipment expenditures. We often make several recommendations based upon the
amount of money the customer wishes to spend.
3. Findings
We have provided services, similar to those we
provided HCFA, to hundreds of companies, and are intricately aware of the
“industry standard” state of security. During our tenure providing security
services to HCFA, we found both extremely positive and disturbing issues.
3.1 Positive
3.1.1. There is a healthy
approach to security from HCFA management. Whereas many other organizations
believe that all security problems can be solved by writing a policy, HCFA has
taken significant steps to not only inscribe the virtues of security, but to
ensure they practice what they preach.
3.1.2. When we first arrived at
HCFA, we found them to be operating with significant and obvious
vulnerabilities. These problems were fixed within hours of our reports. Over
the course of the years, HCFA has become significantly more secure than the
industry standard.
3.1.3. Beyond simply patching
vulnerabilities that were found, HCFA has made significant efforts to find the
systemic causes of their vulnerabilities and fix them wherever possible.
3.2 Negative
3.2.1. Contractors
By far, HCFA’s biggest security problems have been
the direct result of the action or inaction of contractors. In general, we have
found HCFA’s contractors to be outright obstructive to providing sound
security. Compounding these errors was HCFA’s inability to catch or prevent
them.
a.
HCFA
lacked the technical oversight of their contractors to verify the contractor was
actually implementing the security measures they claimed. The managerial
oversight had no ability to ask relevant questions.
b.
HCFA’s
contracts had no mention of security expectations of a contractor. As a result,
the contractors were free to implement (or not implement) any measures they
felt as appropriate, regardless of HCFA’s requests.
c.
We
discovered during our first test that a HCFA contractor ignored change control,
bypassed the firewall policy group, and installed his own filter rules directly
onto HCFA’s primary firewall without anyone’s knowledge. These filter rules
made the entire HCFA network vulnerable to a variety of serious attacks. After
bringing the firewall problem to HCFA’s attention, the contractor was directed
to remove the rule and instructed about the use of change control. One year
later, we tested and found the contractor installed the same rule again without
HCFA’s knowledge.
d.
On
several occasions, we witnessed HCFA contractors argue against improving
security stating that changes HCFA asked for were “difficult” or “impossible”
when, in fact, they were not.
e.
During
our architecture review, we discovered that the HCFA contractors responsible
for network operations could not provide a complete list of all network
connections external to HCFA. In fact, we spoke to over a dozen groups, and
each would make us aware of another undocumented connection from HCFA to
another organization.
f.
Contractors
have made extremely poor password and configuration decisions, violating the
most basic security principals and completely invalidating other security
measures put into place.
3.2.2. IBM
HCFA relied on IBM
to provide secure network connectivity (via a product called “SecureNet”) to
MDCN partners as well as for both external and internal WWW servers. We were
contracted to evaluate the architecture and determine potential risks.
During a meeting with HCFA management (up to CIO
level), IBM’s security staff and management responsible for the HCFA contract,
and ourselves, we were told by IBM that we didn’t need to test because they had
taken every imaginable security precaution. They described how:
-
Administrators
can only connect from a physically secure administrative network
-
WWW
administration is done through an encrypted management connection
-
Patches
are installed immediately after a vulnerability is announced
-
They
would be happy to share their firewall’s access control lists with us
-
They
perform penetration testing every week
-
They
have a custom designed IDS along with 24/7 response
-
The
firewalls only allow WWW access through
-
Upon extensive questioning from ourselves and HCFA’s
CIO, it we learned from IBM that:
-
Administrators
can also dial-in from home into a generic “SecureNet” modem bank, that all
other customers use, and administer machines.
-
WWW
administration can be encrypted, but they haven’t enabled that feature, and
probably won’t because it’s difficult to do so.
-
Patches
are only installed when an administrator gets around to it, which is usually in
a “week or two”.
-
They
would not share their access control lists because, “If EGS found a
vulnerability in HCFA, they would find a vulnerability in all IBM customers.”
Because HCFA relies so much on the security of IBM
to provide everything from secure connectivity for the MDCN to managed web
hosting, we proposed performing three distinct tests:
1)
External
test against the web servers hosted by IBM
2)
Tests
from a HCFA partner connected to the MDCN directed at HCFA and other partners
on the MDCN.
3)
Tests
from a non-MDCN customer of IBM directed towards HCFA.
It took IBM and
HCFA a year of negotiation to come to terms to allow just the external test
against the web servers. We were given several severe restrictions that made
the results of the test unrealistic. Specifically:
1)
We
were not allowed to test the firewalls or any other infrastructure on the IBM
network. They did provide us with an extremely limited subset of filter
rules that IBM said were installed on their firewalls.
2)
We
were not allowed to touch any IBM system other than HCFA’s web server. This
included administrative systems, other customer servers, or any infrastructure.
3)
We
were not allowed to route traffic through any other IBM network.
These restrictions meant that we could only test the
controls in place on the web server. We could not check for configuration
errors in access control lists, vulnerabilities in firewalls or routers, or
transitive trust issues (i.e. if we can break into the IBM administrative
network, or another customer’s web server, what can we do then?).
In the end, the restrictions ended up being
irrelevant. Using an extremely old, very well known vulnerability in the WWW
server software, we were able to gain access to HCFA’s web server without any
more technical expertise than it takes to point and click. Because of the way
HCFA’s web server was configured, and an error made in the firewall rules set
up by IBM, we were then able to access HCFA’s internal network resources. IBM’s
other claims were then shown either false or useless:
-
If they performed a penetration test every
week, they would have discovered this blatant vulnerability
-
They
provided us with the IDS logs collected during the course of our attack, and we
had gone completely unnoticed, despite us making no effort to hide our tracks.
-
The
firewalls allowed not only WWW access through but also another protocol that
allowed us unfettered access to HCFA’s internal network.
3.2.3. Third Party trust
HCFA has a need to connect with a variety of
insurance companies, doctors, and so forth. Network connections were provided
both through the MDCN and by direct connectivity to other companies. These
connections were configured such that there was no protection of either HCFA
from the company or the companies from each other. Essentially, HCFA was
trusting these companies completely. As a result, HCFA is subject to whatever
security policies and protections are in place by the trusted company. So, if
HCFA trusts company A and company A trusts company B, then HCFA trusts company
B. Without any control over or auditing of the partner’s network, HCFA should
not trust that it’s secure.
In addition to threatening HCFA, there is a
potential for these competing insurance companies to use HCFA as a means to
attack one another. HCFA provides the unsecured communications mechanism, and a
company simply uses that to get into another’s network.
4. Recommendations
4.1. Technical Oversight
HCFA is lacking the specially trained personnel to
oversee their and their contractors’ activities and verify the work for
security consistent with policy and best practices. This position should be
solely technical—it should not have any policy development duties associated
with it. The position should be independent of any contractors and should be
associated with the security policy group. Essentially, the role is to be the
“security implementer” of the organization.
The goal is to have an independent and informed
person who can ensure that the security of the organization is not simply some
high-level goals or a policy on paper. In HCFA’s case, such a person would have
prevented the vast majority of vulnerabilities introduced by either HCFA or
HCFA’s contractors. Beyond our experience with HCFA, such a position is direly
needed in most organizations.
4.2. Third-Party Verification
It should be unacceptable for service providers to
certify themselves as secure. Recently, it’s become popular for service
providers to get an outside certification of their networks or services and
provide that as evidence of their security. In our experience, these too are
insufficient, as the certifiers do not reveal their methodology or extent of
their certification.
Any vendor of network services to HCFA should
readily accept 3rd party verification of security and have regular
testing a part of their contract performance requirements.
4.3. Security Specified in Contracts
The security
expectations and requirements should explicitly be laid out in contracts with
network service providers. Without such clauses, HCFA was essentially powerless
to require the use of broadly accepted industry security standards.
4.4. More Testing is Required
Security is such a
complex field, with new vulnerabilities being discovered daily, that it’s
impossible for the average Information Technology professional to keep up. As a
result, it’s necessary to independently verify the security features of
everything from applications to WWW servers to networks and to do so on a
recurring basis. In HCFA’s case, thoroughly testing the security of the MDCN is
critical to its continued operation and information integrity. As it stands,
there’s no way to know if it’s really secure or not. Whenever a new application
or service is provided to the public, a new network connection is established,
or a new modem installed, these need to be tested for proper operation.
5. Conclusion
We believe HCFA
has done more to identify and remedy security problems than is common. Despite
this, they have experienced a substantial set of serious vulnerabilities over
the course of our provision of security services to them. Their reliance upon
contractors to operate makes them particularly susceptible to the types of
vulnerabilities we have described within this document.
There is always
more work to do, however. Security is not a one-time fix—it’s something that
must be integrated into the business of an organization. It must be continually
reinforced, reanalyzed, and redesigned as circumstances dictate. New services,
applications, and networks need to be tested before deployment.
|
