Matasano Security Recommendation #001: Avoid Agents

Thomas Ptacek | December 8th, 2006 | Filed Under: Defenses, Uncategorized

Alan Shimel at StillSecure advocates for a variety of agent-based security systems. Rather than walk into the swamp of a security product marketing debate, I’m going to use his post as an opportunity to make an unambiguous recommendation.

Summary of Recommendations

Enterprise security teams should seek to minimize their exposure to endpoint agent vulnerabilities, by:

  1. Minimizing the number of machines that run agent software.

  2. Minimizing the number of different agents supported in the enterprise as a whole.

Background

Endpoint agents are programs that run silently in the background, usually as Windows Services or Unix daemons, which communicate back to a central management system. Well known examples include:

  • Systems Management (BMC Patrol, CA Unicenter, Microsoft MOM)

  • Antivirus (McAfee, Symantec)

  • Patch Management (Novell ZenWorks, SDS, BigFix)

  • Data Leakage Prevention

Agent-based architectures are a severe security risk. Risk is amplified as more agent-based products are deployed. In enterprises with pervasive agent deployments, attacks on agents are more threatening than attacks on underlying operating systems.

Rationale

Vulnerabilities

Agent-based architectures share a tendency towards unusually complex and hazardous attack surfaces:

  • Listening Network Services on Agents

  • Listening Network Services on Management Servers

  • Client of Agent Service on Management Server

  • Confidentiality and Integrity of Agent/Server Protocol

  • Web Application on Management Server

  • Javascript on Browser Client of Management Server

  • Listening Network Services for Management Clients on Management Server

  • Middleware Frameworks and RPC

  • Client of Management Server Service on Agent

  • Display Logic for Agent-Sourced Data on Management Client

  • Confidentiality and Integrity of Client/Server Protocol

  • Databases

Often, the complexity of the attack surface in agent-based solutions is concealed by turn-key installation and management features. Servers may use embedded-but-listening database servers such as MSDE/Jet, or tunnel DCOM or CORBA in HTTP POSTs.

Virtually every one of these attack surface features is recognized by the industry as a difficult security challenge independently. Deployed in concert, safety concerns are multiplied.

Threats

The deployment circumstances of agent-based products make them an obvious and lucrative target for attackers:

  • Agents tend to be installed en-masse. Attacks that offer uniform compromise of all installed agents provide attacks with thousands of hijacked machines.

  • Even in the absence of an exploit that compromises agent software directly, it is impractical to ensure the security of thousands of endpoints. But every machine running an agent must be secured if the management components are to be shielded from attacks.

  • In a majority of surveyed agent-based systems, compromise of a single management server allows code execution on every agent, exposing the enterprise to a single point of failure.

  • Agent implementations are often substantially homogenous, even across operating systems, enabling uniformly effective attacks against desktops, Windows servers, and Unix servers.

  • Workstations of management operators are high-value IT targets, and compromised agents can inject poisonous data to exploit a myriad of clientside and XSS-style attacks to hijack their machines.

Empirical Evidence

Matasano has surveyed a broad array of different agent implementations, and conducted in-depth penetration tests of more than $4Bn/USD of shipping product and more than ten different vendors. In only one case did a vendor survive a penetration test without a “game-over” vulnerability that would have transformed a deployment into a latent botnet; that vendor had spent tens of millions of dollars over the past year to institute a security development lifecycle and had repeatedly audited its agent-based product.

In almost every other case, we found no evidence to suggest that these high-risk products had ever endured any outside security testing. Classes of vulnerabilities uncovered included:

  • The full gamut of C/C++ runtime vulnerabilities, including simple stack overflows.

  • Metacharacter bugs, including protocols with Unix backtic command execution.

  • Undocumented and unauthenticated protocol commands that enabled command execution or agent reconfiguration.

  • Untested custom encryption protocols, including unprotected key exchanges, insecure cipher modes, replay attacks and crypto bypass attacks.

  • Trivial XSS and SQL injection vulnerabilities that could be triggered from any installed agent.

Matasano presented an overview of these findings at the Black Hat Briefings in August of 2006.

Mitigating Factors

Enterprise security teams required to support agent-based software should prioritize the following objectives for 2006/2007:

  1. Move high-value assets off of servers that must run management agents.

  2. Move high-risk agents, such as laptops and Windows desktops, to seperate management domains with seperate management servers and seperate operations teams.

  3. Obtain detailed documentation for all the protocols used by agents, including the security countermeasures employed by those protocols.

  4. Aggressively filter agent protocols, with host-pair specificity, at every network security border where such filtering is practicable.

  5. Isolate agent management deployments to individual network segments or security domains.

  6. Enable SSL protocol options and institute SSL client certificate access control to ensure all protocol participants at least bear a signed certificate.

  7. Downselect candidate agent management applications to those that have undergone documented third-party security testing.

Conclusion

Agent-based architectures are incredibly convenient and can be a significant cost-saver for IT operations teams. Security teams should assume they already support an unweildy variety of agent software platforms, and should assume that they work under substantial organization pressure to support more agents. It’s therefore critical that security teams crystallize a strategy to minimize the security impact these risky products present.

In all circumstances, enterprises should seek to minimize the number of agent installations within their enterprise.

In all circumstances, enterprises should seek to minimize the number of different agent-based vendors their enterprises must support.

Agent-based software should be treated as a high-risk target for attacks. Agent software warrants intensive security testing and analysis and rigorous access control.

Viewing 26 Comments

    • ^
    • v
    Thomas- there is nothing here I disagree with. I don't advocate a large number of agents, I just think they are a fact of life in IT today though. Even some of the bigger guys like McAfee, who claim they are putting everything on one agent, are only creating an "Uber-agent" with 6 or 7 sub-agents operating underneath.
    Specific to my article, I was just saying that to perform a pre-admission NAC test, sometimes for a variety of reasons, agentless is not going to do the job. At that point, a decision has to be made regarding the risk to use an agent (and all the risk that entails) or do the best you can agentlessly and maybe use some reduced access to mitigate risk. At the end of the day it all comes down to your risk threshold.
    • ^
    • v
    Enterprise security teams, who are your buyers, should have as a primary objective "minimize the number of agent installations and the number of agent vendors supported". I'm pretty sure these aren't primary goals today. They should be.
    • ^
    • v
    Hey, I got an idea. How about we put all the security functions into a single, hardened agent to reduce the number of agents running to as few as possible?

    Wait, we already did that with Blink Professional. V3.0, which comes in the next few weeks, consolidates about 10 different agents into a single agent, hardened from exploitation and managed using a secured architecture.

    Thomas is right in the principle of 'less is more' when it comes to agents. I like the idea of having as few agents as possible running, but with systems management, software distribution and the need to secure the underlying OS and applications, it becomes a need to balance risk against functionality.
    • ^
    • v
    If you don't like agents, would you advocate scanners or sneakernet? (Sarcasm off)

    I can't disagree that customers should look at reducing the number of agents, since Big Fix produces an agent that can (and does) manage just about anyone else's software. And we do that with ONE agent, not by adding slave agents, etc... For example, we will take care of code and sig updates for I think 6 or more AV vendors.

    Any further rebuttal would require some sort of architecture documentation, which I'm still behind on.
    • ^
    • v
    I have several friends at agent vendors. All of them like their agents, and agree that everyone else's are terrible. I'm willing to concede the point if you're willing to concede that our recommendations are good.

    I'll reiterate that we haven't looked at BigFix. BigFix might have excellent security. But if it does, it will be an anomaly.
    • ^
    • v
    Ross: I didn't just say "run fewer agents". I also said "use fewer agent vendors". I see those as two significantly different, equally important recommendations.

    I wouldn't be even close to surprised to find that eEye has the strongest agent implementation on the market; you have an excellent team. But you should be able to document what you did to secure Blink.
    • ^
    • v
    I don't know if the other agents are bad, I'm taking your word for it. I haven't done any real competitive analysis. Sure, many of your recommendations are great, some don't apply, depending on the design.

    I especially like the recommendations to not make programming errors and create bad protocols.

    Oh wait, the sarcasm got turned back on there again for a sec, sorry. ;)
    • ^
    • v
    One of the most broken systems we looked at had smart programmers and had a team that included the designer of a well-regarded authentication protocol.

    How do you know what applies to you or doesn't apply to you unless you've tested? And if you've tested, can't you just explain what you tested, how you tested it, and what it means for the security of your design?
    • ^
    • v
    Classes of vulnerabilities uncovered included

    agreed. this stuff is scary. i take it you're not talking about munin, nagios, or net-snmp agents.

    also did you miss Ross Brown's point about consolidation of agents or did you choose to not respond to it yet on purpose?

    for the client-side, can't we load up netcat to bind to a tcp or udp port on localhost and send enterprise traffic to it (make the management station setting in the client equal to 127.0.0.1), which then pipes to stunnel? and then stunnel sends the traffic to the real management station? and then the management station listens with stunnel and sends the traffic on localhost to the real management application?

    there's a certain `consolidation of agents' in a nutshell.

    maybe it would be a good idea to write the gluelayer that is netcat/stunnel, make it work well with windows, and document it and release it to the public? this would secure the traffic on the wire.

    for now, i'll call it simply, "gluelayer", since it doesn't exist.

    but there is still a problem of the listeners on the enterprise mgmt apps. if you can't shut them off easily (note that i don't trust firewalls), maybe you could completely intercept them assuming they don't bind to every address+socket and also use SO_REUSEADDR, SO_EXCLUSIVEADDRUSE, or enhanced socket security. hijacking a server socket is as simple as binding to a specific IP address (again, gotta love that netcat). another gluelayer project, but this one doesn't work in every situation and has a lot of requirements and dependencies.

    i hate to use the words "layer-3 switches at the access-layer and firewall every port", but i don't trust software firewalls (however, http://force.coresecurity.com is NOT bad).

    but COME ON, TOM. you know that MITM is a much worse attack than attacking the listener since ARP poisoning is so prevalent still. and don't tell me you didn't teach your kids how to use Cain (http://www.oxid.it) yet, because anybody can use this stuff. maybe i should provide some links to DHCP snooping, IP/MAC source guard, and dynamic arp inspection (Cisco's DAI).

    oh crap i just said Cisco, nevermind everything i said. make sure to also set port-security to 1 static and 1 dynamic per port as well to prevent CAM table overflow. and 3,000 other "also's / as well's".

    in reality though people - somebody has already thrown ettercap and openssl into post-phatb0t code (no wait, openssl was in the original phatb0t). downgrade attacks (i could use netsed for this) work great as well. i'm sure Matasano uses more advanced techniques than just these. they probably read chapters 4 and 5 of "hunting security bugs (Microsoft Press)".

    oh and i spelled phatbot wrong. it doesn't have a zero... that was for random self-post-bumping.

    your strategies are great! i guess future recommendations you might want to incorporate would be using consolidated agents, implementing DAI in the network, and/or possibly moving to IPv6.
    • ^
    • v
    somehow the expression 's/strategies/mitigation strategies' got out of the last sentence and into my name as an "i". i'm sure you're all doing your mental calculations / parsing on that and i appreciate. i blame trackpads.
    • ^
    • v
    So here's the long, drawn-out version of my response:
    http://ryanlrussell.blogspot.com/2006/12/nothin...
    • ^
    • v
    You CAN hide your agent traffic in SSL tunnels, and that might not be a bad idea, but remember, one of the central problems with this architecture is that no matter what you do to provide access control and protect the protocols, you have 1000+ machines with legitimate access to the underlying protocol.

    It's not practicable to keep 1000 diverse machines secure. Forget it. It's not going to happen. Now you're in a spot where, when any one of those machines happens, everything you've attempted to do to protect the management server is nullified.

    "It's the application, stupid!"
    • ^
    • v
    And of course there's the information leakage that occurs when the agent tries to phone home to its central server no matter what network it is on or whether the central server can be reached or not. Or worse, just using Karma at that point to spoof the central server....

    You know, the irony is that there are other impacts as well. BigFix's agent may be able to manage other software but it has a massive hit on battery life due to the way it is constantly hitting the disk. It also listens all the time and hence is available no matter what network I'm on.

    I can't speak to Blink 3.0 but I can observe that there is more than just having a kick-ass agent, you have to have a decent central console among other things. Though I have to admit, I have greater hopes that Blink would not have the typical vulnerabilities than other agents.

    McAfee's collection of agents take up more RAM between them than many applications. I can always tell when McAfee is updating because it pegs my processor and kills all response. Now they are trying to integrate it with ePO which just makes me feel all warm and fuzzy...
    • ^
    • v
    Reality is that until we go back to a thin-client architecture, systems and security management agents are required. No other way to do all the things IT needs to do to the end-points, from a security or an operational perspective. Of course it is in the best interest of a network security company to demean agents, just as it it would be in the best interest of an agent-based vendor to cry foul of all the failues in network security.

    You need both. Anyone who tells you otherwise has an agenda. I agree that organizations need to limit the amount of agents deployed, there are far too many fighting for control of system resources. But this doesn't mean you do not need any.

    So the advice is really to eliminate redundant and insecure agents and realign disparate agents into a common management infrastructure.

    Advising organizations to place a high-value assets on syst3ems with no agents it not good advice.
    • ^
    • v
    BTW - Why don't you list the software you tested and found to be insecure or not list any at all. This is kind of like me saying "penetration testing companies are a serious risk to your intellectual property and there is lots of evidence of compromised or stolen code, or vulnerablities found and made public without the research firms following responsible disclosure. Examples of penetration testing companies are Matasano, Symantec." It just seems really disingenuous.
    • ^
    • v
    I can't list the software we tested without listing the software that was vulnerable, because all of it was vulnerable (with one easy-to-deduce exception). We haven't yet agreed on a way to release information about vulnerable vendors before they've patched.

    I'm sorry it seems disingenuous. It isn't.
    • ^
    • v
    Hey Tom,

    Well written. You are not coming to market soon with a product that will prove to be a panacea to this issue, are you?

    -al
    • ^
    • v
    Nope. There is zero chance that agents are going away. There is, at this point, zero notion of doing an agent product here.

    We just don't think people are thinking about agent security enough.
    • ^
    • v
    Hmm, how is it not obvious? Given a choice of achieving THE SAE without an agent or with an agent, who would say "yep, I wonna run another piece of vulnerable software on all of my X,000 servers and X0,000 workstations, etc..."

    However, the sad truth is that sometimes you cannot do it without an agent or you can but you'd open a bigger-than-agent security hole in the process...
    • ^
    • v
    hm ok.. and how are these third party agents any different from your run-of-the-mill OS service?
    In the end its just the same thing and you don't want redundant, complex and bloated services that provide a larger attack surface either right?
    • ^
    • v
    Very few OS services establish transitive trust relationships amongst every machine running the service.
    • ^
    • v
    three words: Pass the hash
    ok, ok, lets not be picky... Berkeley r*
    DNS, NFS, NDS, AD...
    So what's my point? Agents are not an intrinsically bad idea, the design (and implementation) of their current incarnations are. All these technologies were thought off following the paradigms of the previous decades: client/server architectures, centralized control on star-topology or hierarchical networks, monolithic code, centralized databases, etc.
    Distributed databases, mobile code, reputation-based systems and P2P networks are all alien concepts (or at least "interesting research ideas") to the current vendors of agent-based products, if you add to that software bloat, agent complexity and poor security development practices you can easily justify aversion to all agent-based technology but I would not dismiss all of it just because all current, well-known vendors lack the imagination and guts to build things differently and show that they are forward-thinking organizations, at the possible expense of cannibalizing their business.
    • ^
    • v
    With respect to the vast knowledge about systems and security that is collected here, there is a difference that is overlooked in this discussion. From the perspective of an ITSec manager, I've found that when something goes wrong with an agent solution it is his problem, whereas with an appliance solution, it is the vendor's problem.

    It's a matter of what the culture of the company is. Government tends towards appliance solutions because the security group can use it to establish a beachhead in the data centre and extend their influence outward over the various IT silos by offering a service.

    Smaller companies, particularly ones with large mobile sales forces will go for the agent solution because they can actually manage the security of the laptops via the agents, and they have the authority to get the agent on all the machines. This is exceedingly difficult in government systems, though it is occasionally done.

    The minutae of different sectors comes into play to determine which solution is more suitable to them, however, few will ever make the decision of which technology to deply based on the esoteric (albeit important) arguments raised in this discussion.
    • ^
    • v
    I wonder if you guys or anyone has any input into the security of open source agent based configuration management applications like cfengine and bcfg2? Or even, how they stack up against these proprietary ones in terms of what they achieve? Having no hands on experience with proprietary ones I would be curious in knowing...
    • ^
    • v
    And you guys were getting so close. Well, it's a brave new world.

    Be seeing you....
    • ^
    • v
    Not sure what you meant by that...

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus