TODO Group on tooling we can share across many of our peer open source program offices,” he said.
Yahoo helped make Apache Hadoop the cornerstone of the big data revolution when it took the early code and created a team around it to help it scale to internet-scale. They agreed to publish it all as open source. When the need for real-time processing came to the forefront, Yahoo created S4 and open sourced it too, but then discovered Storm was just published too, and it looked more promising. The team ditched their own code and put their efforts into helping make Storm even better.
“We applied to Apache Storm what we learned from Hadoop and S4,” Yehuda said. “Our goal was to make it great, even though it kind of competed with our own first stab.”
Storm is a success today, and the company runs it alongside Hadoop to power many of its products. They added machine learning and high-scale data serving capabilities by adding Vespa Engine, to their platforms, and then published that too. And they helped other machine learning projects scale better too, all by publishing open source.
“We’ve leveraged our expertise with Storm to help both Caffe and TensorFlow achieve better scalability. We don’t own these solutions exclusively. Rather we share our code and help others – all the while we get to leverage our expertise to build one of the industry’s most scalable platforms for our use,” he said. “This saves us money while making us a fantastic place to work on projects that impact hundreds of millions of people.”
The program office worked on strategy, legalities, and execution of these and similar projects. Leveraging open source licensing and processes effectively was a key element throughout. Now as Oath, this work continues and expands.
Yehuda cited three key lessons he learned managing an open source program:
These resources were created in partnership with the TODO Group: the professional open source program networking group at The Linux Foundation. A special thank you to Pam Baker for writing assistance and the open source program managers who contributed their time and knowledge to making these comprehensive guides. Participating companies include Autodesk, Comcast, Dropbox, Facebook, Google, Intel, Microsoft, Netflix, Oath (Yahoo + AOL), Red Hat, Salesforce, Samsung and VMware. To learn more, visit: todogroup.org.
“There are so many edge cases that come up – partnerships, acquisitions, unclear contract terms – and we simply need to be open to learn, explore, and come up with an answer to every open source related question. But the most rewarding part of my job is when people tell me they joined our company because they knew about our open source friendly culture. You know, we’re always looking for open source talent, and I’m hiring into the program office.” added Yehuda.