Overview:

This is our initial benchmarking report for Digital Aero Twin Technology. It was completed as part of the USOPC Tech and Innovation Grant. Since the technology behind AiRO is used across different speed sports, a range of poses representing those sports was chosen, not just cycling. We will continue benchmarking our approach across specific test cases and applications and document our research in future reports.

Goals / Approach:

Our goal is to make sure our riders are faster on race day. It is critical that the recommendations made by AiRO are transferable to the real world. Additionally, we want to be fast, easy to use and affordable, because we believe that the exploratory, wide-scale testing, AiRO unlocks can provide a significant additional performance gain and that more affordable testing can reach more people.

To specify our accuracy needs, and consider any tradeoffs between accuracy, speed and ease-of use, we focused on what is required for AiRO to be useful. To be faster on race day, athletes need to select the best position. This leads to the following order of priorities:

The software needs to rank positions correctly. Given two positions with different drag, the software needs to accurately recommend the one with lower drag. The ability to correctly rank positions is more important than getting the exact numbers right, but of course the ability to rank positions correctly depends on the magnitude of difference between them. Our goal is to be able to discern positions that could also be discerned by an expertly executed wind tunnel or drag test on an experienced athlete. From experience we consider that a best achievable repeatability in wind tunnel tests with live riders is around ±0.5% (1SD), world class is ±1%, experienced riders can achieve 2% repeatability, and novice tunnel testers 3%.
The software should be able to estimate the magnitude of the aerodynamic benefit of a position change to within ±20% of the difference, for a substantial difference of 0.005m2 or more. Implementing a change recommended by the software has a cost, either in the time and effort to adapt to the new position, or the financial cost of acquiring a new helmet or the required fit components to make the new position work. To understand if the effort is worth the benefit, the software needs to get the magnitude approximately right, but it does not need to be exact.
The absolute numbers are less actionable than the relative numbers but might be used to inform pacing strategies and to compare to wind tunnel, track tests or other athletes. Since we don’t simulate the bike and utilize a user-supplied estimate of the drag value of the bike, this puts an upper limit on how accurate we can report absolute values. Different wind tunnels deal with blockage, fixture drag, flow uniformity and turbulence intensity differently, so the absolute numbers are bound to vary from tunnels as well.

We aim to get absolute numbers within 5-10% of tunnel values.

In the long run, AiROs true validation will be if riders use it successfully to find faster positions and go on to win races. And we are off to a great start. Our first partner, the US Speedskating Team Pursuit Team was able to identify hone their technique using AiRO, culminating in a World Championship win in dominant fashion. Another partner of ours was able to make a critical equipment decision based on data that would not have been able to be delivered with conventional testing methods, and velodrome testing later confirmed the advice.

To make AiRO as accurate as possible we looked at academic, peer-reviewed literature and conducted our own wind tunnel benchmarking.

Literature Review and Simulation Approach

After reviewing more than a dozen papers for their methodology, applicability and findings, we settled on two papers as the most valuable to guide our approach.

“CFD Analysis of cyclist aerodynamics: Performance of different turbulence modelling and boundary-layer modelling approaches.” By Thijs Defraeye, Bert Blocken, Erwin Koninckx, Peter Hespel and Jan Carmeliet was published 2010 in the Journal of Biomechanics. It includes a wind tunnel validation on a 1:2 scale model, high resolution pressure measurements for correlation and studied the effect of varying CFD modeling parameters on simulation accuracy.

“CFD simulations of cyclist aerodynamics: Impact of computational parameters” by Thijs van Druenen and Bert Blocken was published in 2024 in the journal of Wind Engineering and Industrial Aerodynamics it includes a thorough literature on the topic of CFD simulation around the human body, followed by a detailed parameter analysis that was benchmarked against expertly executed wind tunnel tests.

The key findings in both papers were that well executed CFD studies can find good agreement with wind tunnels, but the exact selection of simulation parameters, turbulence and wall modelling approaches, matters.

Simulation approaches that selected unsuitable turbulence models, or too coarse mesh parameters can be off by as much as 25%, rendering the value of those simulations minimal or even counterproductive.

The SST k-ω model, without wall models, and appropriately selected wall parameters matched the absolute wind tunnel measurements within 6%. Models with additional terms dealing with laminar-turbulent transition can further reduce the measurement error and reach mean errors of 0.9%. Both reported the absolute errors relative to wind tunnel tests.

In developing our simulation approach for AiRO, we first replicated the most accurate simulation recipe outlined in the paper, then tuned the simulation for speed and affordability. We did this by reducing the numbers of iterations required to convergence by utilizing a good initial estimate of the flow field, tuning relaxation factors, and locally adjusting the mesh resolution on the surface and in the volume. We also take care to select the most performant hardware solution for our simulation problem and optimized our code for the specific hardware.

Currently, we have chosen the k-ω model over the T-SST, and increase reported drag by 5%, since most literature and most of our own testing indicated a consistent under-reporting of drag by this simulation approach by that magnitude. (See next chapters)

We made the decision to forego a transition model since the k-ω model has been validated within dozens of papers, while the T-SST approach is more novel, and has less track record. Tuning the additional parameters related to transition adds additional uncertainty. We intend to pursue a T-SST model once we can validate the approach in additional wind tunnel tests and offer it at a speed and cost that would work for our target users.

Wind Tunnel Benchmarking

In Spring 2024, we tested 30 positions in the wind tunnel and in AiRO to get an initial indication of the agreement between the wind tunnel and simulation. We used the same approach to create our Digital Aero Twin as is available to our users. Since the AiRO team is serving more speed sports than cycling and triathlons, we chose to represent a range of positions, representing cycling, speedskating, running and skiing. The athlete was standing in the test section, no bike was present. We plan to do more cycling specific validation tests in the future.

The tests were conducted at A2 wind tunnel. The test was wearing a whole-body skintight skinsuit with textured arms and legs.

The results of this test can be seen below:

Drag values of wind tunnel testing and CFD compared.

Correlation between CFD and Wind Tunnel Test

Wind tunnel and CFD positions showed a correlation coefficient of R^2 = 0.9987.

After accounting for a systematic scaling factor, the mean variance of the two datasets is 2.3%. Some of the disagreements between wind tunnel and CFD come from inaccuracies of the CFD approach, some of the inaccuracies of the wind tunnel. For the wind tunnel, the main source of error is random variation based on the ability of the test athlete to hold the position.

If we assume a world class testing accuracy of 1% for our test athlete, this would indicate an accuracy of 2% for AiRO. If we assume our tester met repeatability values more in line with experienced, but not world class testers, AiROs random error would be around 1%.

Our validation analysis revealed a systematic scaling difference of 24% between wind tunnel measurements and CFD simulations, with CFD consistently predicting lower values. This scaling factor remained consistent across tests, as evidenced by a very high correlation coefficient (R² = 0.9987) between the datasets. The strong linear relationship indicates that while absolute values differ, relative trends are preserved with high fidelity.

On the same day, we conducted testing with an athlete for whom we possessed track performance data. This comparative analysis showed that the wind tunnel measurements diverged from track data by a similar magnitude (~20%) as observed in the CFD-to-tunnel comparison. This consistency across multiple reference standards suggests the presence of a systematic scaling factor rather than random measurement error and that this effect is related to the tunnel testing.

Additional validation

Additional validation was conducted against proprietary data that cannot be publicly disclosed due to confidentiality agreements. These supplementary tests, which included 3D scans, wind tunnel tests and CFD simulations of athletes, with, and without bike showed consistent performance patterns with the publicly presented results above. The mean variance observed across these proprietary datasets was within similar ranges to those reported in the Wind Tunnel Benchmarking section, but with substantially lower systematic scaling difference of around 5%. While detailed results cannot be shared, these additional validations further support the tool's reliability across a broader range of conditions."

Conclusions and Discussion

Based on our research and data provided, AiRO can be used to discern between different positions with similar accuracy as experienced wind tunnel testers. As computers continue to become faster and calculation approaches continue to improve, the simulation accuracy is also expected to improve further. The scaling difference between our tunnel testing and CFD is unexplained and will be investigated further to increase the confidence in the reporting of absolute CdA values, but the strong linear relationship and high correlation still implies a valid test.

This validation paper focused on the statistical aspects of validation when used as intended. The design decision and underlying technologies used for AiRO dictate limitations of what currently can be simulated. Notably, AiRO can not account for varying skinsuit roughness, fabric seems, textile wrinkles or hair. For a complete and updated list of limitations please consult the limitations section on our website.

We will continue researching and publishing validation reports as we pursue our mission to bring accurate, affordable and easy to use aero testing to our customers.

Benchmarking Report #1

Limitations