EdgeQ Technologies Private Limited Logo

EdgeQ Technologies Private Limited

Big Data Developer

Posted 2 Hours Ago
Be an Early Applicant
In-Office
New York, NY, USA
85K-160K Annually
Senior level
In-Office
New York, NY, USA
85K-160K Annually
Senior level
Develop and maintain scalable big data applications, ETL/ELT pipelines, and real-time streaming solutions. Use Spark, Hadoop, Kafka, SQL, cloud platforms, and databases to integrate, transform, and process structured and unstructured data. Optimize performance, scalability, reliability, and data quality while implementing monitoring and security controls. Collaborate with engineers, scientists, and analysts, troubleshoot processing issues, and document technical solutions.
The summary above was generated by AI
Big Data Developer

Job Title: Big Data Developer
Job Locations: Austin, TX; Dallas, TX; Seattle, WA; Chicago, IL; New York, NY
Job Type: Full-Time
Experience Required: 2–8 Years
Work Mode: Remote / Hybrid / On-site, depending on employer
Estimated Annual Salary: $85,000–$160,000

Job Overview

We are seeking a skilled Big Data Developer with 2–8 years of experience to design, develop, and maintain scalable data processing systems. The ideal candidate will have strong programming skills and hands-on experience with big data technologies, distributed computing, cloud platforms, and data pipeline development.

Key Responsibilities
  • Design, develop, and maintain scalable big data applications and processing pipelines.

  • Build ETL/ELT workflows to process structured and unstructured data.

  • Develop data processing solutions using Apache Spark, Hadoop, and related technologies.

  • Write optimized SQL queries and data transformation scripts.

  • Integrate data from databases, APIs, and external sources.

  • Develop real-time data streaming solutions using Apache Kafka or similar technologies.

  • Work with AWS, Microsoft Azure, or Google Cloud Platform.

  • Optimize data processing performance, scalability, and reliability.

  • Collaborate with data engineers, data scientists, analysts, and software developers.

  • Implement data quality checks, security controls, and monitoring.

  • Troubleshoot data processing issues and maintain technical documentation.

Required Skills
  • Proficiency in Python, Java, Scala, or SQL.

  • Experience with Apache Spark, Hadoop, Hive, or Kafka.

  • Understanding of distributed computing and big data architecture.

  • Hands-on experience with ETL/ELT pipelines and data integration.

  • Knowledge of relational and NoSQL databases.

  • Familiarity with cloud data platforms and data warehouses.

  • Experience with data modeling, performance tuning, and data quality.

  • Strong analytical, problem-solving, and communication skills.

Experience Levels & Estimated Annual Salary

ExperienceEstimated Annual Salary (USD)2–3 Years$85,000–$105,0003–5 Years$100,000–$130,0005–8 Years$125,000–$160,000

Salary estimates are indicative and vary by employer, location, industry, and technical expertise.

Educational Qualifications
  • Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field preferred.

  • Equivalent professional experience, relevant certifications, and demonstrated technical skills may be considered.

Preferred Qualifications
  • Experience with AWS EMR, Azure Databricks, or Google Cloud Dataproc.

  • Familiarity with Apache Airflow, Delta Lake, or cloud data warehouses.

  • Knowledge of real-time streaming, data lakes, and distributed storage.

  • Experience with Git, Docker, CI/CD, and automated testing.

  • Understanding of data governance, security, and privacy standards.

Benefits
  • Competitive salary and performance-based incentives.

  • Health, dental, and vision insurance, where offered.

  • Paid time off and company holidays.

  • Professional development and certification opportunities.

  • Opportunities to work on large-scale data engineering projects.

How to Apply

Interested candidates can submit their updated resume highlighting relevant big data technologies, technical skills, certifications, and project experience.

Equal Opportunity Statement: We welcome qualified applicants from diverse backgrounds. Candidates must meet the applicable US work authorization requirements for the position.

Similar Jobs

One Month Ago
Remote or Hybrid
United States
170K-240K Annually
Senior level
170K-240K Annually
Senior level
Software
Leads the design, development, deployment, and operation of scalable machine learning solutions and high-performance microservices. Builds real-time inference systems using Triton and TensorRT, guides distributed cloud architectures, participates hands-on in coding and reviews, and mentors engineers. Drives technical strategy, operational excellence, system resilience, and business-wide innovation across generative AI, graph machine learning, and big data initiatives.
Top Skills: Amazon SagemakerAWSAzureBig DataCi/CdDistributed SystemsDockerDocument DatabasesFastapiGCPGraph Machine LearningKubernetesMicroservicesNoSQLPythonPyTorchRestful ApisTensorrtTriton
One Month Ago
Hybrid
Senior level
Senior level
Financial Services
Lead development and operation of scalable ETL/data transformation pipelines using Spark and data lake technologies. Design, automate, monitor production workflows, improve cost and reliability, mentor junior engineers, and contribute to platform/SDK infrastructure and CI/CD while enforcing secure, AI-assisted engineering practices.
Top Skills: Apache CassandraApache HbaseSparkAWSAzureAzure Data FactoryBitbucketCloudwatchConfluenceDatabricksDelta LakeDockerEc2ElasticsearchEmrGitGitGlueGrafanaHadoopIamIcebergJavaLambdaPrometheusPythonS3ScalaSpring Boot
2 Months Ago
Hybrid
Senior level
Senior level
Financial Services
Lead an agile engineering team to design, develop, and operate secure, scalable applications. Provide technical guidance, write and review production code, drive AI-assisted engineering adoption, influence product design, and promote firmwide SDLC practices using cloud-native, microservices, and Big Data technologies.
Top Skills: AngularAWSBitbucketCi/CdGitJ2EeJavaMavenMicroservicesPythonReactRelational DatabasesRest ApiShellSparkSpring BootSpring CloudSQLUnix

What you need to know about the NYC Tech Scene

As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.

Key Facts About NYC Tech

  • Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
  • Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
  • Key Industries: Artificial intelligence, Fintech
  • Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
  • Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
  • Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account