Senior Spark Data Engineer - Remote in US
We're seeking a passionate Senior Data Engineer to help us design and build our “big data” infrastructure, data aggregation and ingestion platform. This platform powers our proprietary pricing algorithm and data analysis. You will integrate data from various resources, and manage full data lifecycle (ETL). You will write complex queries on that data, making sure it is easily accessible, and works smoothly. Ultimately, you will be optimizing the performance of our company’s big data ecosystem.
- Design, architect, build and maintain big data infrastructure and tools
- Automate ingestion large amounts of data from numerous sources
- Develop jobs and queries to solve data mining problems
- Write reliable and efficient programs to handles a broad set of big data use cases, including log analysis, MLS data analysis, operational business metrics and machine learning
- Collaborate with talented data scientists to implement predictive models
- Build optimized ETL jobs to power front-end applications
- Data qualification, verification and validation
- BS or MS in Computer Science, Statistics, Mathematics or equivalent
- 5+ years of software development experience and comfortable learning new technologies and tools.
- Strong hands-on expertise with Spark, building distributed streaming and batch processing data pipelines
- Programming proficiency in Go, and at least one of Python or Nodejs
- Strong SQL knowledge (Mysql or Postgres), familiarity with techniques to identify slow queries and debugging (We run on AWS RDS)
- Knowledge and experience with ElasticSearch
- Experience working in the AWS data ecosystem (S3, RDS, EMR, Lambda, Redshift, MQs, Kinesis)
- Familiarity with basics of ML algorithms, concepts in entity resolution and feature extraction
Bonus points for knowledge of:
- Real estate markets and MLS (and tax) data
- Predictive modeling
- Scikit-Learn, R, TensorFlow, Pandas, NumPy
What we can offer you:
- An amazing opportunity to be an integral part of building the next multi-billion dollar consumer brand around the single largest purchase of our lives
- Competitive cash and equity compensation, full medical, dental, vision benefits, 401k, flexible work schedule, unlimited vacation (2 weeks mandatory) and sick time and are open to where you live and work.
We have offices in New York, San Francisco and Atlanta and are a distributed company with employees in 13 different states. This is an amazing opportunity to be an integral part of building a multi-billion dollar consumer brand in an industry that is long overdue for a new way of doing things. You will be working with a passionate, mission-driven team that is disrupting the status quo.
Knock is an Equal Opportunity Employer. Individuals seeking employment at Knock are considered without regard to race, color, religion, national origin, age, sex, marital status, ancestry, physical or mental disability, veteran status, or sexual orientation.
Please no recruitment firm or agency inquiries, you will not receive a reply from us.