Build and configure a government Databricks workspace, develop ingestion pipelines from enterprise systems, and prepare data for reporting. Implement Unity Catalog governance, metadata, auditing, and lineage; configure MLflow workflows; assess security controls for sensitive data; and create architecture documentation, data dictionaries, and operating procedures. The role also requires integrating legacy systems, optimizing Delta Lake pipelines, and supporting long-term platform sustainment.
Responsibilities
Peraton Overview
The Databricks Data Engineer will be responsible for the hands-on build-out of a Government-owned Databricks workspace and the data ingestion/integration work needed to consolidate agency business system data. This includes designing and implementing data pipelines from core enterprise systems, implementing Unity Catalog for data governance, configuring MLflow for machine learning workflows, and developing comprehensive documentation to support sustainment beyond the pilot.
This role is 100% Remote.
Other Responsibilities Include:
- Build out and configure a dedicated Government workspace within the existing Databricks environment
- Design and implement data ingestion pipelines from core agency business systems including financial, HR, CRM, and ITSM systems
- Leverage native/built-in connectors where source systems support them; design custom integration approaches for legacy systems
- Normalize and prepare ingested data within Databricks for consumption by downstream visualization/reporting tools
- Implement Databricks Unity Catalog for centralized data governance, metadata management, active auditing, and end-to-end lineage tracking
- Review current configuration, assess security controls for CUI/PII/PHI/financial data, and implement improvements
- Develop comprehensive "as-built" documentation including physical/logical architecture diagrams, automated data dictionaries, and SOPs
- Document data sources, integration methods, and data lake architecture decisions to support sustainment
Required Qualifications:
- Min 8 years with BS/BA, Min 6 years with MS/MA, Min 3 years with PhD
- US Citizenship
- Active DoD Secret clearance
- 2 years working on the Databricks platform
- Strong proficiency in PySpark, Spark SQL, and Python for large-scale data processing and pipeline development
- Hands-on experience with Unity Catalog administration, including metastore management, access policies, and data lineage
- Experience with pipeline orchestration tools (Airflow, Databricks Workflows)
- Experience with Delta Lake, schema evolution, time travel, and optimization techniques
- Experience integrating heterogeneous enterprise systems, including legacy/custom integrations
- Familiarity with cloud platforms (Azure/AWS) and infrastructure-as-code practices
- Understanding of data governance principles, compliance frameworks, and CUI/PII handling practices
- Familiarity with financial, HR, CRM, or ITSM data structures
- Git/CI-CD pipeline experience
- DoD 8570 certification
Preferred Qualifications:
- Databricks Certified Data Engineer Associate or Professional certification
Peraton Overview
Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day, our employees do the can’t be done by solving the most daunting challenges facing our customers. Visit peraton.com to learn how we’re keeping people around the world safe and secure.
Target Salary Range$112,000 - $179,000. This represents the typical salary range for this position. Salary is determined by various factors, including but not limited to, the scope and responsibilities of the position, the individual’s experience, education, knowledge, skills, and competencies, as well as geographic location and business and contract considerations. Depending on the position, employees may be eligible for overtime, shift differential, and a discretionary bonus in addition to base pay. EEOEEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.Similar Jobs
Aerospace • Information Technology • Security • Cybersecurity • Defense
Lead database engineering for a federal species-data integration program in a hybrid AWS cloud environment. Responsibilities include harmonizing schemas, documenting source-to-target mappings and lineage, enforcing metadata and data-quality standards, implementing encryption and NIST-aligned security controls, building authoritative source registries, optimizing performance, and leading production migration, validation, monitoring, and operational documentation. The role requires Databricks or equivalent lakehouse expertise, ETL programming, federal governance knowledge, and database migration leadership.
Top Skills:
SparkAws GovcloudDatabricksDelta LakePythonScala
.jpg)
Cloud • Information Technology • Software • Business Intelligence
Designs secure Azure data pipelines and analytics foundations using Databricks and ADLS Gen2. Responsibilities include modernizing Oracle workloads, developing medallion data models, implementing data quality and monitoring practices, integrating curated datasets with Power BI, and establishing secure Azure Government identity, network, storage, and access patterns. The role also requires documentation, operational support, stakeholder collaboration, and performance optimization.
Top Skills:
Adls Gen2AzureAzure Data FactoryAzure DatabricksDelta LakeOraclePower BIPythonSQLUnity Catalog
Big Data • Cloud • Analytics • Consulting
Build scalable ETL/ELT pipelines in Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, Workflows, and Unity Catalog. Ingest batch and streaming data across medallion architecture layers, develop transformations and data quality rules, and implement orchestration, monitoring, alerting, and automation. The role requires cloud data engineering, data modeling, performance tuning, CI/CD, and strong knowledge of lakehouse architecture, governance, and distributed systems.
Top Skills:
AirflowSparkAuto LoaderAWSAzureCi/CdData GovernanceData WarehousingDatabricksDatabricks WorkflowsDbtDelta LakeDelta Live TablesDimensional Data ModelingGCPGitLakeflow ConnectLakehouseMedallion ArchitectureMetadata ManagementPysparkPythonRagSpark SqlSQLStructured StreamingUnity Catalog
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
