When was the last time you got paid $40,000 for a few days of work? That is what happened last year to Russian independent security researcher Andrey Leonov, who discovered that if you upload a specially constructed image file to Facebook, you can make Facebook's internal servers, nested deep within their firewalls, run arbitrary commands to expose sensitive internal files in a way that could easily lead to a data breach.
Facebook was lucky that this exploit was discovered by an ethical hacker and not a criminal, and gladly paid him one of the highest bounties ever published for a security bug disclosure.
The bug that Leonov discovered was quite embarrassing, as it revealed that Facebook neglected to update the image processing libraries on its servers. This was the case even though the vulnerability, dubbed Imagetragick, was discovered five months earlier, was widely discussed in tech press and recognized as a high risk threat, and security patches for affected operating systems were made available immediately(1). Last year, it took us 45 minutes to patch our systems against this bug, on the day it was announced, from the moment we found out about it: Imagemagick among other tools is used within some of our image processing pipelines, and we closely follow security announcements about these tools. This is not bragging - we’ve had cases as well where minor, less publicized issues, lingered for weeks before being patched. It does go to show that even one of the most admired software engineering organizations in the world can have its bad days and neglect a trivial patch. On the other hand, Facebook's willingness to publicly discuss and publish this vulnerability is admirable, and promotes the security of the web at large. Stories like this help raise public awareness of the risks in not properly maintaining your servers and software.
How do organizations protect themselves against such risks?
There are several technical and organizational procedural controls that are required for a development organization to stay on top of security risks. In the Facebook case, Leonov collected "low-hanging fruit" via a well-known, published bug. But what about the countless unknown bugs that lurk within the multitude of libraries, software packages and services that are required for an advanced image and video processing system? What about other file formats such as PDF and SVG, which both deserve a special place in the vulnerability hall of shame?
Organizations employ security teams that audit internal software libraries, 3rd-party tools and services, and take care to regularly patch operating systems and software stacks. However, media processing pipelines are especially sensitive to patches and upgrades that change the functionality and resulting outputs, so they are often 'frozen in time' to maintain the same outputs and not break the pipelines.
At Cloudinary, we employ the following principles to guard against risks:
- Assign responsibilities. Information security is considered a strategic issue, and is the responsibility of an executive management team member, coordinating the organization-wide efforts.
- Create, implement and enforce company wide procedures. We created and maintain a set of security procedures, compatible with the widely acclaimed ISO 27001 information security standard. These procedures apply to every level of company activity, from hiring procedures, to writing code.
- Get help. We employ a third-party consulting company specializing in information security standards and procedures to create, maintain and validate enforcement of the controls mentioned above.
- Monitor. No system is free of bugs, even if everybody in the dev org is a security expert and you run the best static code analysis and network monitoring tools. We hire external penetration testers to probe our APIs and web applications regularly. Our systems are patched automatically via daily updates whenever possible. Our security team subscribes to the CERT mailing lists in the countries where we have offices, and gets alerts on new vulnerabilities found in the core libraries and operating systems we use. We also regularly pay (albeit smaller that $40,000) bounties to independent researchers who report security issues in our systems. Watch this space for a future announcement of a public bug bounty program - there are several popular services that provide an infrastructure for such programs.
- Focus. As a SaaS/API provider, the philosophy of using best-of-breed external providers, where available, is ingrained into our culture. We prefer to focus on our own unique software offering than develop internal systems replicating functionality available elsewhere, so we use API and SaaS providers (after vetting them for security) whenever it is possible. Every line of code your programmers write, or library they use, is a potential bug, and using properly vetted providers can help you outsource non-core functionality to experts whose job is to focus on it.
Is this article designed to sell you on using external services for non-core functionality?
Yes, it absolutely is!
In the build vs. buy dilemma, security considerations have pluses and minuses for both sides. One important thing to consider is that security breaches are an eventuality, not a probability, as your website or app becomes more important and popular. If someone penetrates your systems via a malicious image, video or file upload - whose internal network would you rather be breached - yours or the SaaS provider's? If Facebook had used an external provider for image processing, it would save them from having to maintain, patch, continuously monitor and test their image processing pipelines, and dealing with implications of a hacker snooping around their internal networks. Plus, they would get more image and video processing features than they would ever consider developing in-house, allowing them to experiment with new functionality without having to invest development time and the efforts required to make these capabilities secure.
Footnotes (1) According to Leonov's blog, Facebook's un-patched image libraries were compounded by a misconfigured firewall that allowed internal information to leak out to the internet via DNS tunneling.
