A prominent automotive business had a common problem among those active in the mobility-tech industry: handling and making sense of large volumes of high-speed telemetry data efficiently and effectively without sacrificing cost economics and reliability. The rate of more than 120 samples per minute per vehicle overburdened traditional on-prem infrastructure. The answer lay in an AWS cloud solution for big data based on streaming and workflow.
1. High-Velocity Telemetry
Data streams from the connected passenger and electric cars were then sent via a managed Kafka service. It served as the foundation structure for data ingestion. It can be noted that it followed a large-scale setup, like the petabyte-sized data lake provided by Toyota Connected, involving Amazon Kinesis Data Streams and AWS Lambda services for unprocessed sensor messages. Unoptimized partitioning and grouping eliminated issues with small file writes.
2. Scalable Distributed Processing with Spark on EMR
The analytics layer relied on Amazon EMR clusters and Apache Spark for big data cleansing, aggregation, and computation of ranking, streak, and performance values. Based on cost‑optimized EMR practices, spot instances were used for running hourly jobs, auto-scaling of clusters based on workload, and columnar storage in Parquet format. Cost savings and faster data processing with EMR clusters were achieved with 3 times faster data processing and a 30% reduction in EMR costs.
3. Automated Workflow Orchestration
Apache Airflow on EC2 managed daily and hourly jobs without any need for human intervention. Similar to handling various other automotive projects on Amazon MWAA, Apache Airflow triggered Spark pipes, resolved dependency issues, and maintained a timely computation for driver and performance scores. All these steps ensured zero downtime and 99.9% data availability.
4. Real-time visualization and insights
Processed analytics were indexed on an Elasticsearch platform, and dashboards were created for dynamic observation of driver behavior, sentiment analysis, and overall fleet trends. This service followed tendencies used with deployments on Amazon OpenSearch Service, involving EV data, with ingestion pipes feeding operational dashboards and alert systems. Real-time display capabilities enabled anomalous or trending changes to be addressed instantly.
5. Strategies for Optimizing Cloud Cost
Aside from EMR tuning, the architecture implemented cost-saving variables offered by AWS. Data Lifecycle Policies relocated cold data from S3 to Glacier, reducing store costs six times over. VPC Endpoints decreased NAT gateway costs more than 4×. Data compression before Firehose ingestion decreased costs, aligning with Toyota Connected’s strategy for minimizing $/GB costs. DynamoDB and Lambda memory usage adjustments reduced operational expenses.
6. Scalable Reliability and Resilience
A highly reliable system was ensured by carefully scaling, multi-AZ deployments, and fault-tolerant components. The ingestion pipelines were capable of adapting shard numbers on Kafka depending on traffic, and EMR clusters maintained dynamic scaling on core nodes. Adopting event-driven designs for OEMs such as BMW and Audi, components were made fault tolerant.
7. Integration with Broader Mobility Ecosystems
Moreover, the modular structure facilitated interoperable interactions with third-party mobility solutions and APIs. As with Kafka streaming within EV charging infrastructure, telemetry data could be exchanged with insurance systems for usage-based cost calculation or with intelligent city infrastructures for traffic optimization. Registry systems for schema and data contracts ensured a unified and governed approach to integrations, thus avoiding fragmentation in data product capabilities.
8. Lessons for Automotive Data Engineers
Takeaways would be the importance of partitioning awareness during ingestion, the benefits of Parquet and Spark runtime optimization, and storage costs associated with lifecycle storage rules. Cloud architects would learn again about the importance of scaling computing resources and automating workflows to fulfill SLAs while working at high velocity. Mobility tech practitioners would learn about accomplishing multiple operation and analytic workflows with a single pipeline from Kafka ingestion to visualization on Elasticsearch. By aligning ingestion, processing, orchestration, visualization, and cost optimization into a single, cohesive framework on AWS, they were able to turn their existing analytics solution for connected vehicles into a scalable, resilient, and economically feasible solution and an enabler for future mobility innovation.