Technical Lead

Infinite Computer Solutions · Bangalore

Collected role description

<ol start="5" style="list-style-type:lower-roman"><li style="margin-left:8px">Interaction with business users/stakeholders and requirement gathering, Involved Functional Requirement Specifications from customer and understand the same.</li><li style="margin-left:8px" value="5">Good understanding and real time architecture experience on Hadoop ecosystem (HDFS, Map Reduce, YARN, Hive, Sqoop, Spark) and Azure cloud (ADLS, ADFS, ADF)</li><li style="margin-left:8px" value="5">Should have real time experience by using various file formats (csv, parquet, text file, avro, sequential file)</li><li style="margin-left:8px" value="5">Should have Hands-on experience with hive internal, external tables, hive join techniques, partitions, bucketing and hive optimization techniques</li><li style="margin-left:8px" value="5">Should have hands-on experience in developing Spark applications using Data frame API, Spark-SQL in Data bricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing & transforming the data using Scala and Python to uncover insights into the customer usage patterns.</li><li style="margin-left:8px" value="5">Should have hands-on experience Extract Transform and Load data from sources Systems (On-prem) to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Spark SQL.</li><li style="margin-left:8px" value="5">Should have hands-on Experience metadata driven ingestion pipeline framework by using Azure Data Factory</li><li style="margin-left:8px" value="5">Should have real time experience on implementing spark optimization techniques and adaptive query execution configuration in data bricks </li><li style="margin-left:8px" value="5">Should have good analytics and writing SQL queries to consume data and generate reports using Power BI </li><li style="margin-left:8px" value="5">Added advantage bash script experience with text file processing skills using Regex, Sed, Awk, Grep, curl, etc.</li><li style="margin-left:8px" value="5">Cloudera Manager - Monitor jobs and query performance. customizable dashboard, with the ability to create advanced charts for historical monitoring.</li></ol>- Query on Impala nodes to find all the error (Failed queries, Missing Stats, out of memory exception) - Monitoring entire cluster and find memory utilization based on resource level.<ol start="5" style="margin-bottom:11px; list-style-type:lower-roman"><li style="margin-left:8px">Workload XM - monitor the workload, memory, resource utilization, slow job and failed job, queries in spark jobs and impala queries</li><li style="margin-bottom:11px; margin-left:8px" value="5">Involved in conducting the code review and Results with the Developers</li></ol>Have good exposure to Healthcare domain