BH Landscape
Next week, many of us here will be heading down to Las Vegas for Black Hat. The MSRC, and other teams in Microsoft, have been attending Black Hat for years. In fact, we've been sponsoring the show for the last eight years-the last five as a platinum sponsor. Some might ask why? It's funny, I can actually remember back in my days as an officer protecting networks in the U.S. Air Force, questioning why Microsoft had such a presence at the show. As much as I'd like to say it's because of the weather (after all, most of us are over here in the rainy Northwest), or because it's the largest security conference out there (it's not), or even better, because we so look forward to getting our next Pwnie Award-the truth is it's none of the above. Well, maybe just a bit on the Pwnie. But the reality is that to us, Black Hat has always been a reflection of, and driven by, the community-likeminded people from all walks of life and professions with a shared interest in advancing the state of security. They come together to share ideas, advance thinking, network and collaborate, and ultimately learn from one another. We feel connected to that and always look forward to being a part of it.
So with the show fast approaching, I've taken some time to reflect on where the Microsoft Security Response Center is currently and where we see ourselves going with respect to security. Specifically, I've been thinking a lot about three areas: 1) our work to address vulnerabilities in our software, 2) our work with the security community and 3) our philosophy on vulnerability disclosure. Given the fact that each of these topics have recently garnered interest and fueled discussion in the community and media, I thought I'd share my thoughts.
Vulnerabilities and Time to Fix
Some will say that we take too long to fix our vulnerabilities. But it isn't all about time-to-fix: Our chief priority with respect to security updates is to minimize disruption to our customers and to help protect them from online criminal attackers. These customers own and operate a diverse ecosystem of nearly a billion systems worldwide. It's humbling to think about the responsibility this entails and yet we embrace the challenge. Even in the face of that, our overall track record shows the window of vulnerability is being reduced and we have additional plans to improve.
The Microsoft Security Response Center (MSRC) receives more than 100,000 e-mail messages per year at [email protected] - that's nearly 275 per day or 11 per hour. This is filtered down to approximately 1,000 legitimate investigations per year. Once a vulnerability has been confirmed, a comprehensive examination is undertaken to ensure that the reported vulnerability is addressed, other vulnerabilities that might exist in related code are identified and addressed, and no new vulnerabilities or bugs are introduced during this process.
But why don't we commit to fixed timelines? Because it is important to consider the overall customer risk when focusing on updating software for security issues. Most security updates released by the MSRC will be rapidly deployed to hundreds of millions of systems worldwide helping to protect customers from attacks in a very short timeframe. And the software being updated is being used by hundreds of thousands of applications on all sorts of hardware in all sorts of scenarios. So it is imperative that the update has been rigorously engineered and tested in order to avoid creating any type of disruption to these systems. During this time, the MSRC monitors for signs that the vulnerability, or variants, are being used in active attacks. The MSRC does this by using comprehensive telemetry systems as well as data and information provided by customers and partners around the world, and the rest of the industry. This approach helps Microsoft balance between the potential urgency of releasing an update for a particular vulnerability and ensuring high confidence that the update will address the vulnerability, all of its variants and maintain the functionality and stability that customers expect from the affected products.
Many times the issue that the finder reported is an indication of other similar vulnerabilities in that area of code. And the original issue may not be the most complicated, or even the most likely to get used in attacks. Microsoft tries to address vulnerabilities and all of their variants in as few updates as possible because they cost enterprise customers time, effort and money to re-assess and deploy multiple updates for issues that could potentially be addressed in a single update. The time it takes to complete a comprehensive examination helps to ensure the number of security updates Microsoft releases and needs to re-release is kept to a minimum, thus reducing the costs and potential disruption to enterprise customers' operations. Due to the increase in quality that Microsoft has achieved over the last five years, some enterprise customers deploy security updates with little or no testing, and hundreds of millions of consumers continue to use the Automatic Update client on their systems to ensure that they stay protected automatically.
For the majority of issues, we are able to release high quality and comprehensive security updates to customers well before any indication of attacks, and well before they are disclosed publicly. However, there are exceptions. In some cases attacks result, and when that happens, we have to compress testing to release updates quickly. Also, when there are attacks, we release workarounds in days that can block these attacks even without the updates. Usually these take the form of a "FixIt" that can protect customers with one click or be easily deployed throughout the enterprise.
However, there are cases that take much longer. In fact, last year at Black Hat there was a security event dealing with a vulnerability in a library called "ATL" or "Active Template Library." That issue affected not only multiple Microsoft product versions, but also several 3rd party products and services. It took over a year to coordinate that release, and in the end, even the finders themselves understood and commented that with the complexity involved, taking over a year wasn't unreasonable. When seemingly simple security issues, such as a memory corruption bug, affect multiple different products, the coordination and calibration can drive longer timelines so no product, or customers of those products are left behind. And there have also been cases that are such deep architectural changes that they can take multiple years to fully resolve or may not be able to be resolved in some of our older products. Usually these issues result from new threats emerging that product designs or assumptions couldn't anticipate. Changing those assumptions for products that have been in market for several years does take time and coordination so customers and applications can work effectively with them.
Focusing on resolving security issues has and will always be a priority for us. And work to improve our processes will continue, but we must always strike a balance between timeliness and quality.
Working with the Security Community
The topic of how well Microsoft works with the security community is important to me personally, and to my team. Years ago, this was a very valid concern. I can remember being on the outside of Microsoft and watching researcher discussions noting how Microsoft wouldn't engage or was unresponsive. We've made dramatic changes on this front since the inception of Trustworthy Computing. At Microsoft we recognize, and appreciate, the unique value that security researchers play in identifying issues and helping the entire computing ecosystem improve from a security perspective. We also thank many in the community for their collaborative work over the years, and for nearly a decade we have demonstrated our commitment to working with them in an honest and transparent manner. We may not always agree on the severity and the amount of time it should take to develop and test an update that has to work with hundreds of millions of computers, but we do believe we're fair and open when working with researchers. It's not in our interest or the interest of our customers to behave any differently.
Throughout the years we've seen researchers saying that if vendors really valued their work, we'd compensate them directly for the vulnerabilities they discover. That's a trend that's continued in recent weeks. We absolutely value the researcher ecosystem, and show that in a variety of ways. The most well-known is the fact that we acknowledge the researcher's work in our bulletins when a researcher has coordinated the release of vulnerability details with the release of a security update. And that's just the tip of the iceberg. We also work to make sure we can support the community's development by sponsoring and supporting nearly 50 security conferences in over 20 countries each year.
Probably the community effort that started more of the deeper relationships we've built with researchers is our own little "hacker" conference we host at Redmond each year, called "BlueHat Security Briefings." Launched in 2004, this conference is aimed at bringing Microsoft security professionals and external security researchers together in a relaxed environment to promote the sharing of ideas, social networking and ultimately improving the security of Microsoft products. Key to the success of BlueHat and its benefit to our customers is the direct question-and-answer access that researchers get with the specific owners of the technology they're researching. In many cases, some of our direct competitors have sat on our stage at Microsoft and talked about problems in our products, directly to the folks that develop and manage them. And they've been able to get feedback on their research from the same folks as well.
The Shift to Coordinated Vulnerability Disclosure
If there's one area that has had had staying power in terms of driving polarized debate in the broader security community-as manifested in mainstream and social media this past month-it's in how to disclose vulnerability details. Ideally, updates for those vulnerabilities are available for all customers before details are broadly available. This allows us to protect the end-users because they just get the updates automatically, and large Enterprises can analyze, prioritize and deploy updates to hundreds of thousands of systems quickly. When communication breakdowns and disagreements happen, resulting in vulnerability details disclosed by researchers before we release an update, those details are then used by criminals to attack our customers. The worst situation is when vulnerabilities aren't disclosed to the vendor at all, because then there's very little hope of broad protections ever getting released for all customers.
Because of this range of situations, we also see a range of philosophies. Of course, Microsoft always supported the position that the best way to disclose issues is in a coordinated fashion, where details of the vulnerability are released in conjunction with an update that is broadly available for customers. This is known as "Responsible Disclosure." The term itself can be subjective because if either party doesn't abide by those terms, it is implied that they themselves are "irresponsible." Debate on this very issue of responsibility is understandable; however, it is important to remember that in the end we are dealing with customer safety issues - and we should all take that seriously. It is unfortunate these debates can make us lose focus on what is really important - protecting people using the Internet from harm.
Today, Matt Thomlinson, the general manager of Security at Trustworthy Computing, introduced a new disclosure philosophy Microsoft is adopting called Coordinated Vulnerability Disclosure http://blogs.technet.com/b/msrc/arc...ing-coordinated-vulnerability-disclosure.aspx . Katie Moussouris, senior security strategist on the MSRC Ecosystem Strategy team, provides more information and insight on the necessity of this shift in disclosure philosophy and practice on the MSRC Ecosystem Strategy Team Blog http://blogs.technet.com/b/ecostrat...disclosure-bringing-balance-to-the-force.aspx. You'll see from her post, we're not alone in acknowledging it is time for a change. Other vendors and researchers from the broader community of defenders are supportive and will be instrumental in making this shift a reality. So read the post, provide your feedback and then join us in making this an industry wide shift.
Now back to the catalyst for this post-Black Hat. We're just a few days from the event itself and we'll likely see more themes develop once it kicks-off. But I hope the thoughts I've shared here provide some insights into our point of view on recent discussions in the community.
The realities of today's threat landscape point to a world that has shifted from a variety of participants with various motives to one of two sides-those who intend to harm or commit crime and those who intend to prevent harm and fight crime. As an industry and community, philosophical differences or competition aside, we should be in this together. Our own welfare as individuals and a collective community is at stake with unseen criminals who show no indication of backing down. It's our hope that this effort to shift to a shared responsibility of coordination and collaboration is something that is carried beyond Black Hat as we progress and evolve as a global community of defenders.
Hope to see you at Black Hat!
Mike Reavey
Director, MSRC
Link Removed due to 404 Error
More...
Next week, many of us here will be heading down to Las Vegas for Black Hat. The MSRC, and other teams in Microsoft, have been attending Black Hat for years. In fact, we've been sponsoring the show for the last eight years-the last five as a platinum sponsor. Some might ask why? It's funny, I can actually remember back in my days as an officer protecting networks in the U.S. Air Force, questioning why Microsoft had such a presence at the show. As much as I'd like to say it's because of the weather (after all, most of us are over here in the rainy Northwest), or because it's the largest security conference out there (it's not), or even better, because we so look forward to getting our next Pwnie Award-the truth is it's none of the above. Well, maybe just a bit on the Pwnie. But the reality is that to us, Black Hat has always been a reflection of, and driven by, the community-likeminded people from all walks of life and professions with a shared interest in advancing the state of security. They come together to share ideas, advance thinking, network and collaborate, and ultimately learn from one another. We feel connected to that and always look forward to being a part of it.
So with the show fast approaching, I've taken some time to reflect on where the Microsoft Security Response Center is currently and where we see ourselves going with respect to security. Specifically, I've been thinking a lot about three areas: 1) our work to address vulnerabilities in our software, 2) our work with the security community and 3) our philosophy on vulnerability disclosure. Given the fact that each of these topics have recently garnered interest and fueled discussion in the community and media, I thought I'd share my thoughts.
Vulnerabilities and Time to Fix
Some will say that we take too long to fix our vulnerabilities. But it isn't all about time-to-fix: Our chief priority with respect to security updates is to minimize disruption to our customers and to help protect them from online criminal attackers. These customers own and operate a diverse ecosystem of nearly a billion systems worldwide. It's humbling to think about the responsibility this entails and yet we embrace the challenge. Even in the face of that, our overall track record shows the window of vulnerability is being reduced and we have additional plans to improve.
The Microsoft Security Response Center (MSRC) receives more than 100,000 e-mail messages per year at [email protected] - that's nearly 275 per day or 11 per hour. This is filtered down to approximately 1,000 legitimate investigations per year. Once a vulnerability has been confirmed, a comprehensive examination is undertaken to ensure that the reported vulnerability is addressed, other vulnerabilities that might exist in related code are identified and addressed, and no new vulnerabilities or bugs are introduced during this process.
But why don't we commit to fixed timelines? Because it is important to consider the overall customer risk when focusing on updating software for security issues. Most security updates released by the MSRC will be rapidly deployed to hundreds of millions of systems worldwide helping to protect customers from attacks in a very short timeframe. And the software being updated is being used by hundreds of thousands of applications on all sorts of hardware in all sorts of scenarios. So it is imperative that the update has been rigorously engineered and tested in order to avoid creating any type of disruption to these systems. During this time, the MSRC monitors for signs that the vulnerability, or variants, are being used in active attacks. The MSRC does this by using comprehensive telemetry systems as well as data and information provided by customers and partners around the world, and the rest of the industry. This approach helps Microsoft balance between the potential urgency of releasing an update for a particular vulnerability and ensuring high confidence that the update will address the vulnerability, all of its variants and maintain the functionality and stability that customers expect from the affected products.
Many times the issue that the finder reported is an indication of other similar vulnerabilities in that area of code. And the original issue may not be the most complicated, or even the most likely to get used in attacks. Microsoft tries to address vulnerabilities and all of their variants in as few updates as possible because they cost enterprise customers time, effort and money to re-assess and deploy multiple updates for issues that could potentially be addressed in a single update. The time it takes to complete a comprehensive examination helps to ensure the number of security updates Microsoft releases and needs to re-release is kept to a minimum, thus reducing the costs and potential disruption to enterprise customers' operations. Due to the increase in quality that Microsoft has achieved over the last five years, some enterprise customers deploy security updates with little or no testing, and hundreds of millions of consumers continue to use the Automatic Update client on their systems to ensure that they stay protected automatically.
For the majority of issues, we are able to release high quality and comprehensive security updates to customers well before any indication of attacks, and well before they are disclosed publicly. However, there are exceptions. In some cases attacks result, and when that happens, we have to compress testing to release updates quickly. Also, when there are attacks, we release workarounds in days that can block these attacks even without the updates. Usually these take the form of a "FixIt" that can protect customers with one click or be easily deployed throughout the enterprise.
However, there are cases that take much longer. In fact, last year at Black Hat there was a security event dealing with a vulnerability in a library called "ATL" or "Active Template Library." That issue affected not only multiple Microsoft product versions, but also several 3rd party products and services. It took over a year to coordinate that release, and in the end, even the finders themselves understood and commented that with the complexity involved, taking over a year wasn't unreasonable. When seemingly simple security issues, such as a memory corruption bug, affect multiple different products, the coordination and calibration can drive longer timelines so no product, or customers of those products are left behind. And there have also been cases that are such deep architectural changes that they can take multiple years to fully resolve or may not be able to be resolved in some of our older products. Usually these issues result from new threats emerging that product designs or assumptions couldn't anticipate. Changing those assumptions for products that have been in market for several years does take time and coordination so customers and applications can work effectively with them.
Focusing on resolving security issues has and will always be a priority for us. And work to improve our processes will continue, but we must always strike a balance between timeliness and quality.
Working with the Security Community
The topic of how well Microsoft works with the security community is important to me personally, and to my team. Years ago, this was a very valid concern. I can remember being on the outside of Microsoft and watching researcher discussions noting how Microsoft wouldn't engage or was unresponsive. We've made dramatic changes on this front since the inception of Trustworthy Computing. At Microsoft we recognize, and appreciate, the unique value that security researchers play in identifying issues and helping the entire computing ecosystem improve from a security perspective. We also thank many in the community for their collaborative work over the years, and for nearly a decade we have demonstrated our commitment to working with them in an honest and transparent manner. We may not always agree on the severity and the amount of time it should take to develop and test an update that has to work with hundreds of millions of computers, but we do believe we're fair and open when working with researchers. It's not in our interest or the interest of our customers to behave any differently.
Throughout the years we've seen researchers saying that if vendors really valued their work, we'd compensate them directly for the vulnerabilities they discover. That's a trend that's continued in recent weeks. We absolutely value the researcher ecosystem, and show that in a variety of ways. The most well-known is the fact that we acknowledge the researcher's work in our bulletins when a researcher has coordinated the release of vulnerability details with the release of a security update. And that's just the tip of the iceberg. We also work to make sure we can support the community's development by sponsoring and supporting nearly 50 security conferences in over 20 countries each year.
Probably the community effort that started more of the deeper relationships we've built with researchers is our own little "hacker" conference we host at Redmond each year, called "BlueHat Security Briefings." Launched in 2004, this conference is aimed at bringing Microsoft security professionals and external security researchers together in a relaxed environment to promote the sharing of ideas, social networking and ultimately improving the security of Microsoft products. Key to the success of BlueHat and its benefit to our customers is the direct question-and-answer access that researchers get with the specific owners of the technology they're researching. In many cases, some of our direct competitors have sat on our stage at Microsoft and talked about problems in our products, directly to the folks that develop and manage them. And they've been able to get feedback on their research from the same folks as well.
The Shift to Coordinated Vulnerability Disclosure
If there's one area that has had had staying power in terms of driving polarized debate in the broader security community-as manifested in mainstream and social media this past month-it's in how to disclose vulnerability details. Ideally, updates for those vulnerabilities are available for all customers before details are broadly available. This allows us to protect the end-users because they just get the updates automatically, and large Enterprises can analyze, prioritize and deploy updates to hundreds of thousands of systems quickly. When communication breakdowns and disagreements happen, resulting in vulnerability details disclosed by researchers before we release an update, those details are then used by criminals to attack our customers. The worst situation is when vulnerabilities aren't disclosed to the vendor at all, because then there's very little hope of broad protections ever getting released for all customers.
Because of this range of situations, we also see a range of philosophies. Of course, Microsoft always supported the position that the best way to disclose issues is in a coordinated fashion, where details of the vulnerability are released in conjunction with an update that is broadly available for customers. This is known as "Responsible Disclosure." The term itself can be subjective because if either party doesn't abide by those terms, it is implied that they themselves are "irresponsible." Debate on this very issue of responsibility is understandable; however, it is important to remember that in the end we are dealing with customer safety issues - and we should all take that seriously. It is unfortunate these debates can make us lose focus on what is really important - protecting people using the Internet from harm.
Today, Matt Thomlinson, the general manager of Security at Trustworthy Computing, introduced a new disclosure philosophy Microsoft is adopting called Coordinated Vulnerability Disclosure http://blogs.technet.com/b/msrc/arc...ing-coordinated-vulnerability-disclosure.aspx . Katie Moussouris, senior security strategist on the MSRC Ecosystem Strategy team, provides more information and insight on the necessity of this shift in disclosure philosophy and practice on the MSRC Ecosystem Strategy Team Blog http://blogs.technet.com/b/ecostrat...disclosure-bringing-balance-to-the-force.aspx. You'll see from her post, we're not alone in acknowledging it is time for a change. Other vendors and researchers from the broader community of defenders are supportive and will be instrumental in making this shift a reality. So read the post, provide your feedback and then join us in making this an industry wide shift.
Now back to the catalyst for this post-Black Hat. We're just a few days from the event itself and we'll likely see more themes develop once it kicks-off. But I hope the thoughts I've shared here provide some insights into our point of view on recent discussions in the community.
The realities of today's threat landscape point to a world that has shifted from a variety of participants with various motives to one of two sides-those who intend to harm or commit crime and those who intend to prevent harm and fight crime. As an industry and community, philosophical differences or competition aside, we should be in this together. Our own welfare as individuals and a collective community is at stake with unseen criminals who show no indication of backing down. It's our hope that this effort to shift to a shared responsibility of coordination and collaboration is something that is carried beyond Black Hat as we progress and evolve as a global community of defenders.
Hope to see you at Black Hat!
Mike Reavey
Director, MSRC
Link Removed due to 404 Error
More...
