Team Lead MLOps Engineer in a team of 6 developers for 5 MLOps Streams:

1. RecSys Products - My Team and I launched and participated in ML System Design for many RecSys Business Products with ML, which increased revenue of RecSys Team to third of total revenue of WB:
   • Developed and Launched product Search by Photo V2 using deployed by our team embedding database Qdrant, daily new embeddings updates using deployed by our team Airflow and Triton for inference, that increased the revenue of product 4 times;
   • Developed and Launched product Autogenerated description of product WB card using Mixtral7bx8 in vLLM on 50.000 sellers using SAQ - this feature will be monetized in future;
   • Created Triton Instance with HPA and daily model update with zero downtime for Nearline calculation of User Embeddings in 8000 RPS;
   • Developed and created Dags in Airflow with difficult business logic and large amount of steps, integration connections (more than 15) and MIG for Item2Item Recs
2. Pipeline Orchestration:
   • Developed and launched with my team Airflow 2.x in K8s as Pipeline Orchestrator for more than 200 ML Developers with main killer feature - launching DAGS in different K8s & Spark/Hadoop clusters from Single (!) UI.
3. Online Inference DL/LLM:
   • Developed and standardised process for launching ML/LLM model from DS Experiment to Production using Triton Nvidia Inference Server and vLLM - Time2Market significantly decreased. Launched with my team more than 40 ML models in production: modern CV/Text Embedders, Detectors, OCR and others;
   • Developed and used lots of optimization techniques for online inference: created library Bertolt for DL utilies, such as Model weights conversion: PyTorch, Onnx, OpenVino, TensorRT; Model Analyzer for performance tuning, DALI and HPA for Triton and vLLM;
   • Made lots of research experiments with my team for choosing best framework for LLM inference: custom LLM benchmarks for Triton TensorRT LLM, vLLM, Text Generation Inference and Sglang, vLLM optimizations (Tensorize and others);
4. ML Tracking: Patched open-source code for fileserver in ClearML to make proxy to S3 for artifacts instead of local storage, also created automatic user creation in MongoDB and launched test ClearML Agents. Launched ClearML in production on more than 300 users.
5. ML Tools:
   • Made an major release with my team for Python library MLTool 1.0.0 with 97% coverage, automatic and versioned documentation for library by code and lots of useful features - wrappers for Triton, Airflow, DB Connectors, S3Client[s5cmd], Docker Container Fixtures for integration tests and Quick Insert to Postgres. More than 250 developers are using library and love it for clean code and candy features. We also created a survey for developers to popularize MLTool in quiz-mode - How well do you know MLTOOL?
   • Made and developed with my team lots of repo templates for quick start: launching Airflow DAG in 5 minutes, launching web service in 10 minutes and Python Library distribution using Nexus
   • Standardized process of getting documentation in actual state - made MLOps documentation as point of start for DS developers

I also made what Team Lead should do: comfortable planning process, generated new ideas for tasks, participated in tones of interviews for new MLOps developers (wrote texts for vacancies ), launched Tech Demo inside MLOps Team, participated in code review, demo’s, retro’s and grooming, wrote digest’s of sprint result and lot’s of other things.

Writing code and deployment in production Online / Offline ML/DL services with full software engineering lifecycle using modern stack of technologies

Responsibilities:

• Optimizing code for recommendation services on main page, deploying ML Models as a web service using Triton Nvidia Inference Server (Swin Transformers, VIT SIGLip, YoloV8), writing code and deploy models as a DAG’s (more than 40 pipelines using Prefect)
• Deployed the largest open source LLM Mixtral7bx8, the first LLM in WB using Nvidia TensorRT LLM framework on 2 GPU’s using Distributed Queue Service using Celery and Redis + RabbitMQ + VLM LLaVa deployment
• Data collection from different sources and storing data in analytical and relational databases (Postgres, ClickHouse, Greenplum and S3), writing historical data in DWH using K8S & Prefect;
• Writing own MLOps library on python for common utils tasks (connectors for many databases, S3 loaders and backups, request wrappers) using best practices for python packaging with 100% tests coverage;
• Writing backend and API on python using FastAPI and other services, write monitoring of services metrics using Grafana, Thanos and Prometheus, logs to Kibana + Elasticsearch stack;
• Writing CI/CD pipelines on GitLab; Deployment of other services on k8s using helm charts;
• Writing unit and integration tests, provide load testing, use monitoring and alerting for services;
• Participating in infrastructure planning, demos, code reviewer.

Technologies:

• Python 3.x, Poetry, PDM, Commitizen, Nox, Rye, Asdf, Nexus
• FastAPI, SQLAlchemy, Celery, Streamlit, Triton Nvidia Inference Server, TensorRT LLM Backend, LLMs(Mixtral, Openchat, LLaVa)
• PySpark, MLFlow, Airflow, Prefect, DVC, K8S CronJobs, ClearML
• Docker, Docker-Compose, Gitlab Ci/CD, K8S, Helm Charts, K9S, Vault, Multi Instance GPU (MIG)
• Postgres, ClickHouse, S3, Greenplum, Redis, RabbitMQ, Kafka
• Milvus-on-cluster, Voyager, Qdrant
• PyTest, PyTest-cov, Mypy, Black, flake8, PyLint, Ruff

Responsibilities: Studied cloud technologies for promotion to the role of T2 Big Data Engineer, took advanced certification courses for obtaining certificates from Microsoft and AWS.

Reason to leave: Grid Dynamics closed offices in Russia, unable to relocate.

Technologies: Python 3.x, Amazon S3, Amazon DynamoDB, PySpark, Airflow, GCP