TOSCA Automation Engineer

Azure Databricks Administration & Management:

• Deploy, configure, and manage Azure Databricks workspaces in a scalable, cost-efficient, and secure manner.
• Administer clusters, jobs, notebooks, and workflows , ensuring high availability and performance.
• Monitor and optimize compute resource utilization and autoscaling strategies to improve cost efficiency.
• Manage Databricks Runtime versions, libraries, and dependencies across environments.

Security & Compliance:

• Implement and manage Unity Catalog for fine-grained access control and data governance .
• Enforce Role-Based Access Control (RBAC) and integrate Databricks with Azure Active Directory (AAD) .
• Ensure compliance with SOC 2, HIPAA, GDPR, and internal security standards .
• Set up audit logging, monitoring, and alerting for security and operational insights.

Performance Optimization & Troubleshooting:

• Tune Apache Spark workloads to improve query performance and resource efficiency.
• Analyze and troubleshoot performance bottlenecks in ETL and ML workloads.
• Optimize Delta Lake storage, caching, and indexing strategies for better query execution.

Automation & Infrastructure as Code (IaC):

• Automate Databricks workspace deployment using Terraform, ARM Templates, or Databricks REST API .
• Develop and maintain CI/CD pipelines for Databricks job deployment and configuration management.
• Implement monitoring solutions using Azure Monitor, Prometheus, or Grafana .

Collaboration & Integration:

• Work closely with data engineers, data scientists, and DevOps teams to support data pipelines and analytics workloads.
• Integrate Databricks with Azure Data Lake, Azure Synapse Analytics, and Snowflake .
• Provide technical guidance and best practices for efficient Spark job execution and cost optimization .

Required Skills & Experience:

• 5+ years of experience in Azure Databricks administration and performance optimization .
• Expertise in Apache Spark, PySpark, SQL, and Scala .
• Hands-on experience with Databricks Unity Catalog, Delta Lake, and MLflow .
• Strong knowledge of Azure cloud services (Azure Data Lake, Azure Synapse, Azure Key Vault, etc.).
• Experience in Infrastructure as Code (Terraform, Bicep, or ARM Templates) .
• Proficiency in CI/CD pipeline automation using Azure DevOps, GitHub Actions, or Jenkins .
• Strong understanding of network security, identity management (AAD), and encryption best practices .
• Excellent problem-solving skills and ability to troubleshoot complex Databricks workloads.
• Strong communication and documentation skills.

Preferred Qualifications:

• Databricks Certified Associate or Professional certification.
• Experience with Azure Kubernetes Service (AKS) and serverless computing .
• Familiarity with Kafka, Apache Airflow, and event-driven architectures .
• Knowledge of Python, PowerShell, or Bash scripting for automation.