Open Source Compliance in the Enterprise.)
You’ll also need to have a simple internal approval model for open source use and contributions. Throughout the years, Samsung has moved from highly complex and cumbersome policies to a more simple approach for receiving, reviewing, and approving source code contributions. It’s a function of balance between all parties involved: legal, engineering, and open source. The compromise we have now supports the dedicated open source team which has a blanket approval to contribute to a number of open source projects. This is not the case for other teams who need to get different levels of approval depending on the nature of the code being contributed (e.g. simple bug fixes, code to improve existing functionality, code that offers new functionality, or starting a new project).
You’ll also want to ensure your tools are compatible with the open source development model from the start. Create a setup that fulfills the needs of the open source program office and also meets corporate IT guidelines.
Samsung, for instance, issues Linux devices that work with all of the tools its engineers need to participate in open source development. This environment allows developers to join the team without requiring any major changes to the way they work. They also support work from home and in fact have only very few open source developers in their Silicon Valley offices – all other staff are remote across the globe.
First, a bit of history. December of 2000 marked a major milestone in the history of open source when IBM pledged to spend $1 billion on Linux R&D. IBM was a true pioneer in the enterprise world putting its bet on Linux and open source when very few companies did – and definitely not at that scale. IBM had to learn a lot about working with open source software and the various project communities they were involved in.
That was the starting point for enterprise open source and companies have since followed IBM by the dozens and then hundreds. Yet, thousands of companies are still entering open source as it is becoming the new normal of software development. The question is: How can we minimize the enterprise learning curve and speed up the process of “getting it right?”
The answer is easy, but the application of that answer to specific company culture is the hard aspect. Let’s explore some of the “easy” aspects of learning from over 17+ years of enterprise experience with open source software. You’ll get to deal with the hard aspect of applying those lessons to your particular enterprise environment.
A word of caution: In addition to implementing these practices, you will need to lead a cultural shift from traditional software development practices to a more open and collaborative mindset. Internal company dynamics need to be favorable to open source efforts. As an open source leader inside your organization you will face several challenges in terms of funding resources, justifying ROI, getting upstream focus, etc. These often require a major shift in mindset and a lot of education up the chain.
It takes considerable time to grow internal open source expertise. Hiring key developers is a critical step that allows your organization to quickly gain skills and recognition.
Two or three people are a great start towards making a noticeable impact in a large project such as the Linux kernel, attracting further hires, and allowing enough resources to mentor existing junior developers. (See our guide on Recruiting Open Source Developers.)
The goal is to find people who have enough peer recognition to be influential in the community. There are typically three pillars to this: domain expertise, open source methodology, and working practices.
You also need to align corporate interests with individual interests: it’s very hard to motivate a senior open source developer to work when their personal interests don’t meet with corporate interests in a given project. For example, a Linux memory management expert may not be interested in working on file systems, a corporate priority. Therefore, finding a match in interests is critical for a long lasting relationship.
The core principle for hiring open source developers is to support your open source development and upstream activities. There is also the expectation that they should support product teams in their expertise areas. However, it’s not uncommon for product teams to exercise their influence in an attempt to hijack the time of the open source developers by having them work on product development as much as possible. If this happens, many open source developers will head to the door, seeking a new job that allows them to work on their upstream project before you realize what just happened.
Therefore, it’s important to create and maintain a separation of upstream work and product work. In other words, it’s recommended to provide your open source developers with guaranteed time to meet their upstream aspirations and responsibilities, especially if they are a maintainer. For junior developers or other internal developers who are using open source in product components, such interactions with the upstream community will increase their language, communication, and technical skills. In the absence of such an upstream time guarantee, it’s easy for these team members to be sucked into becoming an extension of product teams, resulting in their upstream focus drying up in favor of product development.
You should also be the upstream partner for product teams; they often feel like they’re living inside a pressure cooker, especially in a consumer electronics environment. They often seem understaffed, lack critical resources to support parallel upstream development, and are under constant pressure for feature delivery within tight schedules. In such an environment, it’s very easy to overlook the benefit of upstreaming in favor of short-term time savings that can unfortunately lead to technical debt that has a higher cost in the long term. Open source teams can help by being a partner that focuses on delivering important code upstream, reducing this technical debt.
Encourage developers outside the open source team to learn from and contribute to the open source community. We provide as much help as we can with upstream code contributions, but our resources are limited, and we don’t always have the deep understanding of products that might be needed to adequately identify where code can be upstreamed. Better involvement in the open source community from teams outside our own allows us to get more important code upstream, and improves our ability to interact with the community.
Grow open source talent in specific technology areas relevant to your products. It’s easy to hire a few resources from outside the company, but there are several limitations to this approach.
The alternative approach is to convert your existing developers into open source contributors via training on the technical domain and open source methodology. These developers can then be paired with a mentor to further expand their skills.
Set up a mentorship program where senior, experienced open source developers provide mentorship to junior, less experienced developers. Typically, the mentorship program would run for 3 to 6 months; during this time, the mentor should supervise the work of the mentee, assign tasks, and ensure proper results. The mentor would also do code reviews for anything the mentee produces, and provide feedback before the mentee pushes the code to the upstream project.
The goal is to increase the number of developers the company has contributing code to the upstream project, and to improve individual effectiveness by increasing the quality of code and the percentage of code that is accepted into the upstream project. Generally speaking, no more than 4-5 mentees should be assigned to a given mentor, and ideally they should work in the same area as the mentor to make code reviews more efficient.
Create an open source developer track in your human resources (HR) system so people hired as open source developers, or those building their open source skills, have a good sense of how their career will progress within the company. Additionally, you should adjust performance-based bonuses to include goals related to open source development work. The metrics by which the performance of proprietary or closed source developers are measured are often different than those of open source developers.
At some companies, there is a clear distinction between open source and non-open source developers. But in many companies the line is much more fluid, depending on its organizational structure and open source strategy. In reality, all modern developers have to work with open source. There are no closed source developers. Rather, there are times their code stays inside the company and times their code is published (contributed to a third party, or published as a new project). Your HR track and incentives should reflect your organization’s unique structure and approach to open source.
Finally, allow a flexible work from home policy for open source developers, regardless of the general corporate policy related to this. Lately, we have witnessed a reverse in work from home policies across companies where many have either banned or created strict limitations to working from home. In the open source world, a work from home policy is almost mandatory because open source experts are located all over the planet, and this policy is often the only way to hire them.
There are operational benefits to a flexible work policy as well. Offering remote work options reduced employee turnover, and “job attrition rates fell by over 50 percent,” according to a study published by Stanford University. And a study by PGI, a leading provider of software services, found that 80 percent of workers reported higher morale when working from home, while 69 percent reported lower absenteeism.
It’s impossible for any company to hire all the senior and most expert developers in a given domain. This concept applies to the Linux Kernel and any other prominent open source project. Therefore, you must have a way for your company to increase the competence of its developers in a given technical domain. In addition to technical training, you’ll also need training to teach the open source development model and the basic concepts of open source legal compliance.
Sample training courses include:
Support your developers to attend and participate in open source conferences and events, including local community meetups, hackathons, and summits. Such participations help them connect at a personal level with their peers, build relationships, have face to face social interactions, and participate in technical discussions that guide the project direction.
If your developers have work that others might be interested in, help these developers prepare content to present. Finally, you can also sponsor events, both big and small, to increase external visibility within the project’s community. These events are also a great venue to look for talent!
Provide a flexible IT infrastructure that allows open source developers to communicate and work with the open source and Linux kernel community without any challenges. Additionally, set up internal IT infrastructure that matches the tools used externally to help bridge the gap between internal teams and the kernel community or any other open source project community for that purpose. Much of this infrastructure will naturally evolve with your organization’s open source culture, but it’s important to be aware of the necessity and plan for its implementation
There are three primary domains of IT services that are used in open source development: knowledge sharing (wikis, collaborative editing platforms, and public websites), communication and problem solving (mailing lists, forums, and real-time chat), and code development and distribution (code repositories and bug tracking). Some or all of these tools will need to be made available internally to properly support open source development. There is a chance this might conflict with existing company-wide IT policies. If so, it’s vital to resolve these conflicts and allow open source developers to use the tools they are familiar with.
These open source practices typically require an IT infrastructure that is free from many standard, limiting IT policies.
“It took us years of constant discussion and negotiation to break from the traditional IT setup into a more flexible environment that supports our open source development. We made it work for us and with enough persistence you also can make it work for your open source team.” – Ibrahim Haddad, Vice President of R&D and Head of the Open Source Group at Samsung Research America.
Create an internal system to keep track of developer contributions and impact. Contributions can include upstream development, supporting product teams, knowledge transfer (mentoring, training), visibility (publications, talks), launching new open source projects, and establishing internal collaboration projects with other teams or groups.
There are several toolkits that help track source code contributions; for instance, The Linux Foundation uses a tool called gitdm, which produces the data reported in the Linux Foundation yearly Linux Kernel report. This can be used to track both individual developers as well as the overall team performance. Individual developers can be tracked for the number of patches they submit, the patch acceptance rate (patches submitted divided by patches accepted), and the type of patch (e.g. if it is a new feature, enhancement of existing functionality, bug fix, documentation, etc.).
Other tools like GrimoireLab can also be used to chart and visualize the metrics you want to track. See the next section on metrics for specific examples of what you should track.
Contribute to and focus on areas that benefit more than one business unit or more than one product. This allows you to provide value and show ROI across multiple business units and increases your chances for more funding and support.
Focus your contributions on upstream projects that would directly benefit the company’s strategy and products. In open source development, it’s very easy to get carried away hopping between different interesting projects. In an enterprise setting where the open source group is considered a cost center, your driving force should be to focus on open source projects that support product development.
Samsung does a yearly review of its product portfolio in an effort be involved in open source projects that are commonly used across as many products as possible. This list is then prioritized based on several factors, and they focus their efforts on the top projects. A methodology that drives your priorities is a great way to stick to what’s important, justifiable, and fundable.
Collaborate with other business units that use the same open source projects in their products. These collaborations can take one or more of many forms:
The goal of these collaborations is to help the product teams understand their needs and fulfill their product goals via open source enablement.
Your company must share information and priorities across different divisions. To illustrate this, assume you are in an open source team and you are requested to support the implementation of a driver, but you are unable to get access to the hardware manual and instructions. This sounds a bit like playing darts with the lights off, and it is. Information sharing is a critical component to successful internal collaborations between the open source teams and everyone else.
Sidebar: Open source at Samsung
Samsung’s Open Source Group (OSG) was established in February 2013 to support two primary functions: the first is to provide open source leadership within Samsung by helping other divisions in the company understand how to participate in and benefit from open source development. The second is to serve as Samsung’s representatives in the wider open source community. The mandate of the team is to focus on enhancing key open source projects and technologies via active contributions to them, and to be actively involved and engaged with various open source organizations and foundations.
Once you start implementing these open source best practices, you’ll need proper open source metrics to drive the desired development behavior. But the traditional metrics often used in product organizations don’t apply in the context of open source development.
For example, tracking the number of changesets or lines of code can be a good metric for open source development impact. But you may have multiple instances of desired functionality being implemented upstream because your open source developers lobby for support from the community. In this case, the number of changesets or lines of code doesn’t matter nearly as much as the technical leadership that team members provide to get code upstream and reduce the company’s downstream maintenance efforts. So the metrics you track should account for both activities.
One of the most basic things to track is the number of commits and lines of code changed over a specific period of time, such as every week, month, or year.
With this data, you can compare contributions from various internal development teams to identify where source code contributions are coming from and help ensure that resources are allocated appropriately.
From here you can create charts that compare various internal teams for their cumulative contributions, percent of total contributions, and the amount of time it takes to get code committed upstream (see the following charts).
Figure 2: Cumulative contributions over time can be tracked to compare internal teams and identify teams that are increasing their involvement in a particular open source community (in this chart, it’s the Linux kernel).
Figure 3: Displaying your company’s contributions as a percent of total over time allows you to identify the teams that contribute the most code.
Figure 4: The amount of time it takes to commit code upstream can be valuable for tracking your development efficiency. This table and chart shows how quickly various teams are getting their code contributed upstream and compares it to the community as a whole.
You can also use these metrics to compare your performance to other companies who are involved in the Kernel ecosystem for instance (Figure 5). This competitive analysis helps you be better informed about the overall developer ecosystem for the project.
Figure 5: Cumulative contributions can be sorted by company to see how your company stacks up against others.
These metrics provide a much better idea of where your strengths and weaknesses are and can help inform your overall development strategy. Tracking your own contributions relative to competitors’, for example, provides valuable information that helps an organization position its products relative to competitors’ in the marketplace.
Figure 7: Projects may publish contribution data independently as well. For example, you can also track contributors to the Linux kernel via The Linux Foundation’s Linux Kernel Development Report and LWN.
The Linux kernel is of strategic importance to Samsung so the company chose to focus its development efforts there, among many other projects. They are now regularly among the top 5 contributors to the Linux Kernel by changesets. And the company is achieving the same progress with several other open source projects it deemed critical to the development of its products.
Being a top contributor isn’t a goal, in and of itself, but rather an indication that the organization’s development efforts are being accepted by the communities in which it participates. As the saying goes, you can either sit at the table as an open source influencer, or be on the menu.
“Being on the menu means sitting and crying in your office about how you need to maintain a ton of code out-of-tree because you can’t merge with upstream. We’d rather sit at the table.” – Ibrahim Haddad, Vice President of R&D and Head of the Open Source Group at Samsung Research America.
Effective open source development can’t be given, it must be earned. This leadership is earned through regular, ongoing participation and contribution.
By following some of the open source best practices set out by the pioneering companies in open source, you can make fast progress toward developing the internal open source expertise you require. You can then leverage that expertise to improve your products and services while reducing code maintenance costs.
This was a collaborative effort spearheaded by Ibrahim Haddad (Ph.D.) who is Vice President of R&D and the Head of the Open Source Group at Samsung Research America. We would like to express his sincere gratitude to Ben Lloyd Pearson (Operations and Strategy, Samsung Open Source Group for the continuous support, reviews and edits (following the open source model) that shaped this article and helped get it to the finish line. Big thanks also goes to Brian Warner (Manager of Open Source Engineering and Strategy, Samsung Open Source Group) and Gil Yehuda (Open Source at Oath) for their review and feedback.These resources were created in partnership with the TODO Group: the professional open source program networking group at The Linux Foundation. A special thank you to Pam Baker for writing assistance and the open source program managers who contributed their time and knowledge to making these comprehensive guides. Participating companies include Autodesk, Comcast, Dropbox, Facebook, Google, Intel, Microsoft, Netflix, Oath (Yahoo + AOL), Red Hat, Salesforce, Samsung and VMware. To learn more, visit: todogroup.org.