![LF](https://d2abrqas732254.cloudfront.net/images/specialists/e48d45b7c838630bd9b37ba144dfda00a85b385a.jpg)
English:
Upper Intermediate
Luciano F.
Vetted by YouTeam
Brazil
UTC -03:00
America/Fortaleza
English:
Upper Intermediate
Experienced Data Engineer proficient in SQL, Python, Spark, Databricks, and Azure Data Services.
• Leading and developed the Data Warehouse Migration project to a Data Lakehouse on Azure, demonstrating leadership and technical expertise. • Implementing data governance improvements, including a Data Catalog service and observability tool. • Reducing the average execution time of the main ELT pipeline in the legacy environment, showcasing optimization skills and the ability to improve existing processes. • Building predictive models for estimating freight trains' transit times and improving forecasts, indicating advanced analytical and modeling skills.
Want to hire this engineer?
Check if Luciano is availableExpertise
Years of commercial development experience
12 years of experience
Core technologies
Other technologies
Project Highlights
Grupo Recovery
Dec `21 - Oct `23
2 years
Data Warehouse Migration project to a Data Lakehouse on Azure with Python and Spark.
Responsibilities & achievements
• I developed and deployed the structural parts. Currently, I am working on data cleaning and transformation with two more data engineers whom I have been training. • Boosted Spark Jobs' performance on Databricks by optimizing queries, data caching, table partitioning, cluster resizing, and query improvements, resulting in a substantial reduction in execution time and computational costs. • Improved data governance in the company's Big Data environment by implementing a Data Catalog service (Unity Catalog), an observability tool for clusters and jobs (Overwatch), and personalized access controls for internal and external data based on users and areas. • Created data quality validation and data cleaning processes that drastically reduced the ticket opening rate related to data quality. • Reduced the average execution time of the main ELT pipeline in the legacy environment from 12 to 7 hours by reformulating the orchestration, tuning SQL procedures, and identifying and removing unnecessary steps.
Dti Digital
Apr `21 - Dec `21
8 months
Predictive models that provide real-time estimates of freight train transit times on a logistics company’s railway network.
Responsibilities & achievements
• Completed a predictive time series analysis project, which delivered a 12% improvement in refractory material consumption forecasts.
Belgo Arames
Nov `19 - Mar `21
1 year
ETL pipeline for steel importation data from external sources and designed data models that integrated with information extracted from the CRM.
Responsibilities & achievements
• Developed a model that identifies possible tax evasion fraud on national steel imports. In the first 6 months, more than 30 confirmed cases of fraud were identified by the model. • Joined the company's Innovation Ambassadors team, becoming the point for data solutions.
Usiminas
Feb `19 - Oct `19
8 months
The model is used to adjust sales strategies, budget definitions, and purchases by the Supply Chain team.
Responsibilities & achievements
• Improved the visibility of growth and fluctuation of steel demand by customers, with a predictive modeling of Machine Learning. • Developed a People Analytics project with the HR team, helping them to identify professional profiles and control the turnover of more than 14 thousand employees.
Education
Higher Education
Agency
400+
GMT-11
Remote
Want to hire this engineer?
Check if Luciano is available