It is no surprise that we see many of the most innovative open source projects (OSPs) incubated within large 'millenial' Enterprises. Giants like Google and Facebook have an unprecedented production volume that cannot be easily replicated. As such, these large Enterprises have very unique problems that need to be solved in creative new ways. Unfortunately for Enterprise software giants like Oracle, IBM and SAP, the real-time world moves too fast for traditional Entperise software consulting. Our young giants like Google, Facebook, Yahoo, Spotify, Instagram, Twitter and Etsy must fend for themselves and innovate internally.
OSPs rely on these large young Enterprises for the unique problem scenarios they uncover, while at the same time the OSPs rely on the startup community to continue innovation in real-time. As the final lifecycle component for a successful OSP, we see large traditional Enterprises the last to adopt, exploiting the learnings from their more agile, increasingly successful competitors.
Open Source Project Lifecycle
We are surrounded by breakout OSP victories: Linux, MySQL, Ruby, Node.js, nginx, OpenSSH, Asterisk; the list goes on and on. With many of the 'big Internet problems' already solved several times over, we see more specialized solutions emerging for our real-time and hyperconnected world.
Let's look at five open source projects that will continue to make waves in 2016.
Released in 2011, WebRTC continues to excite the developer community. WebRTC is the new real-time communication standard baked into next-generation web browsers. WebRTC facilitates peer-to-peer communication across the web, like audio, chat and video. Most importantly, it's plugin free, so no need to download Skype or Flash. At its core, WebRTC provides simple APIs that:
- getUserMedia - access video and microphone data
- PeerConnection - manage real-time connection data
- DataChannels - send ancillary data between connections
What all this means is that application developers can easily pass real-time data between clients without the need for complex filtering and routing of that data. WebRTC has far-reaching applications beyond something like Google Hangout, Talky, or peer-to-peer cams. Segments like travel, health insurance and customer service will all be rapidly changed by WebRTC.
real-time communication with WebRTC
Web security will continue to be a trend so long as we have connected devices, i.e. forever and a day. As 2014 was littered with cyber attacks and massive consumer, government and military data breaches, 2015-16 will see this trend continue aggressively. To get a visual idea of the type of global problem we're dealing with, Kaspersky powers an interactive real-time cyber threat map to illustrate the scale, frequency and scope of existing cyber attacks.
Kaspersky real-time Cyberthreat Map
As the threat landscape deepens, the open source community continues to respond to the problem. The cyber threat arena is very unique in that open source tools can be used for both good and malice; the medicine can also be the antidote. Hackers and security consultants can be using the exact same tools.
Cyber threats are getting so out of control that even the government has stepped in requiring identification for download of Metasploit due to tightening encryption export rules.
Introduction to Metasploit for Penetration Testing and Hacking
Distributed computing continues to grow at a breakneck pace. It seemed like in 2010 only the most brave engineering teams would roll their own distributed cluster. As we look to 2016, it will be commonplace. The tools are now maturing enough to allow just a small team of engineers to run and manage a dynamic, fault-tolerant distributed system.
Released in 2013, CoreOS is an ultra-lightweight flavor of Linux built for large-scale distributed environments. CoreOS acts like a master-less system, where any deployment of CoreOS within the same group will communicate across the group to support routing, failover and other types of distributed behavior. CoreOS is also designed with application containers in mind. By using containers to separate applications from the base operating system, myriad security improvements can be achieved by way of easier system updates.
The key insight in our thesis is that all security boils down to the ability to update software. You can make software hardened and more secure but no software is perfect. We said, let’s build a server that can automatically update itself. That’s very different than the way people think about servers now. If this works, we thought we could unlock a lot of of value, that value being around security, reliability, performance, really everything you get from running the latest version of software. [...] It’s definitely a new way to run infrastructure and that new way will fundamentally impact how companies do it today versus how they will do it in the future.
Intro to CoreOS
Also released in 2013, Docker continues its meteoric rise of adoption. Docker works within a Linux machine by creating lightweight, isolated application containers that can be separately configured, allocated for, and optimized. Docker provides the connection API to each container, allowing the containers to remain indifferent to the operating system.
Introduction to Docker
Because Docker simply and generically abstracts applications from the operating system, it makes distributed systems more portable, with any applications configured for Docker able to run on another server with Docker installed. We also see a great deal of interest in combining CoreOS with Docker, creating lightweight distributed clusters.
When compared with AWS, the same team running their own cluster in a physical datacenter can see cost savings up to 60% and notable performance improvements. Companies like eBay, Gilt, Shopify, Spotify, Google, RedHat and Twitter have all acknowledged the production use of Docker.
ASM.js presentation from
What we have seen to date is native mobile and desktop ecosystems having a clear advantage for building applications whose usability depends on accessing the power of the processor. Applications like Photoshops, Instagram, and audio/video editing applications would be a good example of this. We are used to seeing a clear division between 'native apps' and 'web apps.' E.g. most iPhone apps only perform on the iPhone or iPad, they must be rewritten to function on other platforms.
While we have seen incredible proprietary advancements from companies like Apple and Microsoft, organizations like the W3C and Open Web have always been working toward web standards that elevate web performance near parity with native applications.
The promise with WebAssembly is that our web browsers should be able to run most any application from most any language, at or near native application performance levels. This is huge. If we think of the web browser as the generic 'container' for an application, then these applications can be completely device and operating system agnostic (just like we have today with web applications). While Apple and Android require Objective C or Java, these same applications and many more can be ported for the web and mobile web, to run using the browser as a generic container.
Adding to our important trio of open source projects in the distributed computing space, Google's Kubernetes is the logical progression for orchestrating a large distributed container deployment across a fleet of virtualized or bare-metal servers.
Containerizing the Cloud with Docker on Google Cloud Platform
At the core of the distributed computing problem is communication, status, health and consistency across a cluster of related nodes or containers. Each node must be able to report its status, data or attributes at various levels of granularity. A set of algorithms then act on that node's declaration to determine how to keep that answer consistent with other nodes in the cluster. A single node must also have knowledge of its relationship to other nodes, e.g. leader/follower or master/slave, and said algorithms must be able to dictate the proper process to ensure consistency and integrity.
As expected and in part due to its relation to CoreOS, etcd has received a great deal of attention since its release in 2013. It is also the distributed store behind Google's Kubernetes project.
etcd: distributed locking and service discovery by Brandon Philips
Timeglass / Glass
While time has always been an important function of any development process, it is generally used in the form of personal estimates or man-weeks. In a trend similar to what we've seen with heart rate and distance calculation in the wearables and personal health segments, using actual time writing code as a development metric will become much more common.
Glass is one such project in this space that seems to be gaining momentum.