Last month, SpotXers took the stage in San Francisco for the annual Spark Summit to discuss the SpotX’s use of Spark Streaming applications. Senior Software Engineers Landon Robinson, Jack Chapa, and Ben Storrie shared tips and tricks on navigating Spark applications. Learn more from the SpotX speakers and check out their sessions below!
The Spark Listener interface provides a fast, simple, and efficient route to monitoring and observing your Spark application – and you can start using it in minutes. In this talk, we introduce the Spark Listener interfaces available in core and streaming applications, and show a few ways in which they’ve changed our world for the better at SpotX.
The opportunity to share the feats we’ve achieved with this impressive, open-source technology was the greatest reward of the conference. It was gratifying to have folks approach us after the talks and say things like, ‘this was one of the best talks’ or ‘this is something I can’t wait to go home and try’ — especially because our goal was to create talks that were relevant and applicable to the big data engineering community.
There were a good number of technical questions, which for my part I’d hoped would be the case. I enjoy digging into people’s specific problems and scenarios. I had one person, at the end of the Listeners talk, mention that they’d been attempting to do something like this for months, and that they were elated when they saw our topic on the list of tracks. On a different note, I enjoyed one attendee’s question that started with the comment “this is pretty cool, you guys must have a lot of people working on this,” which for a team as small as ours was a pretty satisfying compliment.
But overall, to be helpful to someone by spending time answering their questions, is very gratifying. That is largely inspired by my first conference experience, a small regional show put on by Hortonworks, where I met Owen O’Malley, founder of HortonWorks, and leader of projects like ORC and Hive. He answered all of my deepest technical questions, which genuinely helped with a long-running problem I was having! So thanks again, Owen!
Landon Robinson
Senior software engineer at SpotX
Spark Summit is a fantastic conference with engaging topics, and opportunity to speak with industry leaders. It was interesting to learn about upcoming features of the platform, as well as how other industry organizations solve similar problems to the ones SpotX faces every day.
We received some questions around how we continue to grow our platform with new Spark features, and how we’ll continue to evolve our solutions. We’re excited to grow and expand our team as we continue to deliver success for our customers.
Jack Chapa
Senior software engineer at SpotX
The Spark Summit was a fantastic time to learn about active development to improve our lives in the big data ecosystem, as well as to learn from other companies using and consuming those improvements. The affirmation in other talks from large companies regarding the technical approach and problem solving techniques we use at SpotX, and the deep dives on the low level core of the stack were encouraging and useful. We also came away with a few ideas of how to improve the lives of the consumers of our data!
The questions we received that were curious if we had a large team in order to implement our solutions probably stood out the most, but we had some great technical questions during the Q&A time, and a few people came up with more after the “official time” was over, and we were able to discuss their questions at length, and discuss implementing them in newer paradigms offered by Spark.
Ben Storrie
Senior software engineer at SpotX
Headaches and Breakthroughs in Building Continuous Applications
At SpotX, we’ve built and maintained a portfolio of Spark Streaming applications — all of which process records in the millions per minute. From pure data ingestion, to ETL, to real-time reporting, to live customer-facing products and features, continuous applications are in our DNA. Come along with us as we outline our journey from square one to present in the world of Spark Streaming. We’ll detail what we’ve learned about efficient processing and monitoring, reliability and stability, and long term support of a streaming app. Learn from our mistakes, and leave with some handy settings and designs you can implement in your own streaming apps.
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
The Spark Listener interface provides a fast, simple and efficient route to monitoring and observing your Spark application – and you can start using it in minutes. In this talk, we introduce the Spark Listener interfaces available in core and streaming applications, and show a few ways in which they’ve changed our world for the better at SpotX.