Leads data architecture for a federal Azure and Databricks AI platform supporting reporting, analytics, and machine learning. Responsibilities include guiding customers and VA stakeholders, mentoring senior engineers, designing lakehouse and warehouse architectures, building metadata-driven ingestion frameworks, governing data with Unity Catalog, enabling reporting and ML workloads, protecting PHI and PII, supporting ATO requirements, and contributing to technical proposals.
BlueFlag is hiring a Principal Data Architect to lead data architecture for a premier data and artificial intelligence (AI) platform at the Department of Veterans Affairs. The platform powers reporting, analytics, and machine learning for Veterans' health and benefits. This remote role is customer-facing every day. You will lead senior data engineers, guide VA leaders through architecture decisions, and help build the platform's future on Azure Commercial with Databricks and Unity Catalog.
We're looking for an engineer first: someone who built pipelines and warehouses hands-on, then grew into leading teams and advising customers. You'll join a friendly, collaborative team doing work that makes a real difference for Veterans and their families.
What You'll Do
- Work with customers daily: lead architecture discussions, brief VA stakeholders, and turn their needs into clear designs and decisions.
- Lead and mentor senior data engineers through design reviews, code reviews, and hands-on problem solving.
- Set architecture and standards for ingestion, storage, modeling, and serving with Azure Data Factory, Synapse Analytics, Databricks, and Delta Lake.
- Architect the platform's move to Azure Commercial, with Databricks as the engineering platform and Unity Catalog governing data, access, lineage, and models.
- Design metadata-driven frameworks that move thousands of tables through batch, incremental, change data capture (CDC), and streaming patterns.
- Serve reporting and machine learning workloads through the Synapse Dedicated Pool, Databricks SQL, Power BI, and model training pipelines.
- Protect PHI and PII through access policy, masking, and Authority to Operate (ATO) support under HIPAA and federal security requirements.
- Write architecture and technical approach sections for proposals.
Why Join BlueFlag
At BlueFlag, we're passionate about leveraging cutting-edge technology to make a real difference. You'll be at the forefront of cloud innovation, working on projects that directly impact people's lives. We offer a high-growth, entrepreneurial environment that values fresh ideas and authentic teamwork.
If you're ready to take your data engineering career to new heights and contribute to meaningful projects that push the boundaries of technology, we want to hear from you. Join BlueFlag and be part of a team that's shaping the future of AI solutions!
Requirements
- 12+ years in data engineering and data architecture, starting as a hands-on data engineer, with 5+ years as a lead or principal architect
- Proven leadership of engineering teams, including mentoring senior engineers and setting technical direction
- A demonstrated history of customer-facing work: leading architecture discussions, briefing executives, and writing for technical and non-technical readers
- Expert, hands-on cloud data engineering and data warehousing on Azure: Azure Data Factory, Synapse Analytics (including Synapse Dedicated Pool), Databricks, Delta Lake, and ADLS Gen2
- Hands-on Unity Catalog design: catalogs and grants, attribute-based access control, row filters and column masks, and lineage
- Design of lakehouse (medallion) architectures, data warehouse models, and metadata-driven frameworks across thousands of tables
- Strong proficiency in SQL and Python (PySpark)
- Experience with CDC, streaming, and incremental load patterns (e.g., Oracle GoldenGate, Azure Event Hubs, Apache Kafka)
- Experience migrating large on-premises data warehouses to the cloud, including parallel runs and reconciliation
- Experience with Git, CI/CD, and infrastructure as code for data platforms (e.g., Terraform, Databricks Asset Bundles)
- Working knowledge of HIPAA, NIST 800-53, and FISMA requirements for PHI and PII
- Bachelor's degree in computer science, information systems, or a related field.
- US Citizen: Must be a citizen of the United States
- Security Clearance: Must be able to obtain a public trust clearance. Must be eligible to work in the United States.
Desired
- Experience with VA health data (CDW, VistA, Oracle Health, HL7, FHIR)
- Experience in Azure Government or other FedRAMP High environments
- Experience with Immuta, Collibra, or Microsoft Purview
- Experience with Databricks SQL, serverless compute, Microsoft Fabric, or Power BI semantic models
- Experience with MLOps and model serving on Databricks
- Databricks or Azure architect certifications
Benefits
- Competitive salary
- Generous annual leave and paid holidays
- Comprehensive group health and dental plans
- 401(k) with company match
- Life insurance and AD&D coverage
- Ongoing training and professional development opportunities
Similar Jobs
Big Data • Information Technology • Machine Learning • Sales • Software • Database • Generative AI
Architects and standardizes cloud infrastructure for data platforms on GCP, including Kubernetes, networking, Terraform, GitOps, CI/CD, observability, performance, cost, security, and disaster recovery. Evaluates emerging data technologies, runs prototypes, guides managed-service connectivity, supports governance and consolidation, and enables engineering teams through architecture reviews, workshops, reference implementations, and documented standards.
Top Skills:
Ai/Ml InfrastructureApache AirflowApache BeamApache FlinkApache IcebergAWSAws PrivatelinkBigQueryCi/CdCloud ComposerCloud SqlCmekConfluentConfluent FlinkData MeshDataflowDistributed Sql DatabasesDnsElasticsearchFeature StoresFinopsGCPGitopsGkeGoogle Pub/SubGraph DatabasesIamKafkaKubernetesLakehouseLlmsMcp ServersMongoDBPostgresPrivate Service ConnectService MeshSnowflakeSolrTerraformVector DatabasesVpc PeeringVpc Service ControlsWorkload Identity
Aerospace • Information Technology • Professional Services • Security • Software
Design and own an event-driven data architecture that moves healthcare claims data through AWS and Snowflake into fraud, waste, and abuse alerts. Establish architecture standards, approve material changes, build complex pipeline components, and ensure scalability, reliability, and compliance with PII/PHI constraints. Collaborate with data science, compliance, technical, and investigative teams while supporting government review timelines. This is a hands-on principal role requiring strong Python, SQL, AWS, Snowflake, and production streaming pipeline experience.
Top Skills:
AWSPythonSnowflakeSQL
Insurance
Leads enterprise data architecture strategy, modernization, and multi-year platform roadmaps. Defines standards for data models, ingestion, transformation, orchestration, quality, governance, observability, and consumption across batch and streaming environments. Provides technical leadership to data engineering and analytics teams, oversees scalable data solutions, mentors technical leaders, and partners with business, engineering, security, and data science stakeholders. Evaluates technologies, guides vendor selection, enables data products, and explores GenAI applications.
Top Skills:
AirflowAmazon RedshiftAWSAzureBigQueryCi/CdClaudeDagsterDatabricksDbtGCPGeminiGenaiGitInformaticaInfrastructure AutomationKafkaOpenaiPythonSnowflakeSQLTivoli
What you need to know about the NYC Tech Scene
As the undisputed financial capital of the world, New York City is an epicenter of startup funding activity. The city has a thriving fintech scene and is a major player in verticals ranging from AI to biotech, cybersecurity and digital media. It also has universities like NYU, Columbia and Cornell Tech attracting students and researchers from across the globe, providing the ecosystem with a constant influx of world-class talent. And its East Coast location and three international airports make it a perfect spot for European companies establishing a foothold in the United States.
Key Facts About NYC Tech
- Number of Tech Workers: 549,200; 6% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Capgemini, Bloomberg, IBM, Spotify
- Key Industries: Artificial intelligence, Fintech
- Funding Landscape: $25.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Greycroft, Thrive Capital, Union Square Ventures, FirstMark Capital, Tiger Global Management, Tribeca Venture Partners, Insight Partners, Two Sigma Ventures
- Research Centers and Universities: Columbia University, New York University, Fordham University, CUNY, AI Now Institute, Flatiron Institute, C.N. Yang Institute for Theoretical Physics, NASA Space Radiation Laboratory
