Classification Optimization

Cost vs VUs and Batch Size Contour Plots

These contour plots visualize the cost per billion tokens per day (1B_cost) as a function of VUs (Virtual Users) and Batch Size for different hardware configurations (hw_type) and image types (image). There are real points, but in-between I

How to Read the Charts:

• Color Gradient: Shows the cost levels, with darker colors representing higher costs and lighter colors representing lower costs.
• Contour Lines: Represent cost levels, helping identify cost-effective regions.
• White Dots: Represent real data points used to generate the interpolated surface.
• Red Stars: Highlight the lowest cost point in the dataset.
• Small Red Dots: Indicate the lowest cost for each batch size.
• Tight clusters: (of contour lines) indicate costs changing rapidly with small adjustments to batch size or VUs.

How to Use:

• Identify the lowest cost configurations (red stars and dots).
• Observe how cost changes with batch size and VUs to optimize your setup.
• Compare different hardware types (hw_type) and image processing strategies (image) to find the best-performing configuration.

Analysis

Overall we can see that nvidia-t4s are more expensive for this cost-ratio and task. We should consider using the nvidia-l4.

We can see a clear winner with nvidia-l4 over nvidia-t4 at this cost ratio. But surprisingly we see default slightly outperform trt-onnx. I think we should be careful not to overfit. These numbers can vary per run, but its good to know that each image can be competitive.

nvidia-t4

• Here we can see that trt-onnx and default both perform similarly but with trt-onnx having a slight edge.
• trt-onnx has a lower overall cost band (611–659) than default (623–676)

nvidia-l4

• trt-onnx has a broad area of relatively low cost and hits a very low floor (~255)
  • This is great since it shows that we get consistently good results!
• default can also dip into the mid‐200s in certain spots, but it has bigger, more expensive areas—especially at lower VUs and batch sizes.
  • This means we need to spend time to optimize it

Conclusion

If I have time, I might analyze the nvidia-l4 with trt-onnx across some different runs. Despite being $1.25 more expensive per 1B requests its a safer more consistent bet IMO.

Best Image by Cost Savings

Chart

• Color = Best Image for that vu/batch_size/GPU
• Size = % cost savings vs. the worst (most expensive) image in that group.
• Small dots dont mean that much, large dots do

Analysis

We can see that trt-onnx is quite a bit stronger in nvidia-l4. There are no significant red dots.

nvidia-l4

• trt-onnx (blue) dominates most points, indicating it's typically the cheaper choice
• At larger batch sizes (right side) and higher VUs (upper part of the chart), you often see big blue bubbles, suggesting trt-onnx can save a significant percentage versus default
• A few red points (i.e., default cheaper) appear at lower batch sizes, but they're less frequent and often show smaller savings differences

nvidia-t4

• There's more of a mix: some points favor default and others favor trt-onnx
• You can see some large red bubbles, meaning default can occasionally produce big savings under certain (VUs, batch_size) conditions
• However, trt-onnx is still cheaper in many scenarios, especially toward higher batch sizes

Takeaways

If you have time/budget, its better to analyze both. You can see that they are close at times. But if you only have time/budget at the current cost ratio consider the nvidia-l4 in this case.

1B Requests Cost vs. Latency

This scatter plot visualizes the relationship between average latency (ms) and cost per billion requests per day for different hardware types (hw_type) and image processing configurations (image).

How to Read the Chart:

• Point Symbols: Represent different hardware + image configurations.
• Color Gradient: Represents batch size, helping to see cost trends across different batch sizes.
• Hover Data: Displays additional details like VUs, batch size, and throughput per second.

Key Features:

• Global Minimum Cost (Red Star): Marks the configuration with the lowest cost.
• Pareto Front (Red Dashed Line + Points): Highlights the most efficient configurations, minimizing both cost and latency. These configurations offer the best trade-offs.

How to Use:

• Find the lowest-cost, low-latency configurations by looking at points near the bottom-left.
• Use the Pareto front to identify cost-effective configurations.
• Compare different hardware and image processing strategies to optimize your setup.

This visualization helps in selecting the best configuration balancing performance (low latency) and cost efficiency.

Failed Requests Check

If all requests were successful, the result should be 0.

Did we try enough VUs?

How do we know that we tried enough VUs? What if we tried a higher amount of VUs and throughput kept increasing? If thats the case then we would see a monotonicly increasing relationship between VUs and Throughput and we would need to run more tests. Lets check this out!

We can check by:

1. Grouping data by hw_type and batch_size to match how we generated the experiments
2. Sorting the data by vus within each group to ensure we get the data in the correct order for our check
3. Check for monotonic increase in throughput_req_per_sec flag the groups that always increase throughput as VUs increase

But how do we know? We can use the slider to check what would have happened if we had not tried past a certain amount. Lets say that if we tried 256 instead of our actual 1024 we would have left some potential on the table, we can simulate this by filtering our runs.

Verification

1. Put the slider at 256 and see that there are a number of scenarios where we should have checked for a higher VU count
2. Put the slider at `1024 and verify that there are no scenarios shown

Are we Accurate Enough?

We shouldn't expect to see significant changes in accuracy. We should see a pretty tight distribution, but there might be some deviation since for lower VUs we wont have as many samples of our 10_000 that we saw with higher VUs.

Here we can see some deviation with a large z-score, but overall its not that big of an absolute devation. These also occur when we have relatively low total_requests which makes sense.

We should worry more if we see major absolute_deviation with higher total_requests. We can see those values here: