52nd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference<BR> 19th
4 - 7 April 2011, Denver, Colorado
AIAA 2011-2095
A Modified Complex Algorithm Applied to Robust Design
Optimization
Johan A. Persson1 and Johan Ölvander2
Linköping University, 581 83, Linköping, Sweden
Today there is a desire to perform optimizations in order to receive optimal system
properties. However, for computationally expensive simulation models, an optimization may
be too tedious to be motivated. This paper proposes a modification of the Complex
optimization algorithm to enable the creation and usage of local meta-models during the
optimization. Its performance is demonstrated for a few analytical problems and a reliability
based design optimization is conducted for an aircraft example.
Nomenclature
Av
βi
g(x1,x2)
k
m
mp
n
pe
pt
Rn
tf
xi
y
ŷ
=
=
=
=
=
=
=
=
=
=
=
=
=
=
maximum opening area of vent valve
the ith coefficient for a response surface
the mathematical function g as a function of the variables x1 and x2
number of points used for complex algorithm
number of samples
mass of piston inside the main valve
number of variables
end pressure in the environmental control system
tank pressure
n-dimensional space
time to fill the environmental system
value of the ith variable
function value
estimated function value from a meta-model
I. Introduction
T
he design process, from product idea to market release, is an iterative process where decisions need to be made
under uncertainties.1 These uncertainties might stem from varying operating conditions, geometrical variations
due to manufacturing tolerances, model assumptions etc.2 In order to see how these uncertainties affect the expected
performance of the system, it is possible to perform an uncertainty analysis. The information from this analysis
enables estimations of failure probabilities and how robust a design is.
The use of an optimization algorithm is an efficient way of searching the design space for the optimal design.3
Therefore, it is tempting to place the uncertainty analysis inside an optimization algorithm to receive optimal
designs. In robust design optimization (RDO), the aim is to receive an optimal design which is insensitive to
variations and uncertainties,4,5 whereas reliability based design optimization (RBDO) aims at a design with minimal
failure probability.5-7
However, numerous simulations of the model are needed to receive accurate results from either RDO or RBDO.
Moreover, an optimization may need hundreds of evaluations of the objective function to converge. For RDO and
RBDO, the objective functions are the statistical properties of the system characteristics and when an uncertainty
analysis needs to be performed during every iteration of the optimization algorithm, the required number of model
simulations is enormous.
1
2
PhD Student, Department of Management and Engineering, johan.persson@liu.se.
Professor, Department of Management and Engineering, johan.olvander@liu.se.
1
American Institute of Aeronautics and Astronautics
Copyright © 2011 by Johan Persson. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.
Consequently, the required time to perform an RDO or RBDO for a computationally expensive model, such as a
large Computational Fluid Dynamics (CFD) model, may be unrealistically long. One way of solving this is to create
and fit a meta-model (MM) of the original model. Meta-models are approximations of the original model which are
more computationally efficient while yielding approximately the same results as the original model. However, it
requires many samples to fit one meta-model so that it represents the original model accurately over the whole
design space.8
Therefore, this study aims at lowering the required number of model calls for an optimization algorithm by
modifying it so it creates and uses local meta-models throughout the optimization. The chosen algorithm is the
complex-RF algorithm found in Ref. 9.
A proposed methodology for performing RDO is described in section II, while section III describes the modified
optimization algorithm. The use of the modified algorithm for optimizing a few analytical and engineering problems
is handled in section IV, and finally section V concludes the paper.
II. A proposed methodology for RDO and RBDO
First, a model of the proposed system is developed. The model might consist of numerous parameters, making
the possible design space large. It is desirable to lower the complexity of the problem by identifying the parameters
which affect the system characteristics most and neglecting the others. One way of revealing the relative importance
of the model parameters is to perform a sensitivity analysis.10,11
Then, the errors and uncertainties in the model and its parameters are estimated. It is desirable to try to fit them
to standard probability distributions, such as the Gaussian or Poisson, which are described with only a few
parameters.6
There exist many methods for estimating how uncertainties and errors in model parameters affect the results
retrieved from simulations of the model and a few are described in more detail in section II-A. The general aim is to
propagate the uncertainties and errors through the model to estimate the resulting uncertainties and errors in the
system characteristics.
The results from the uncertainty propagation will be in the form of statistical distributions for the values of the
system characteristics, enabling decisions to be made based on probabilities. It is then possible to place the whole
uncertainty propagation step inside an optimization algorithm by choosing which statistical entity the objective
function should be based on. For RDO, the objective function is a combination of the mean and variance values of
the system characteristics of interest,4 whereas it may be the probability of failure or success in the case of RBDO.5
A. Propagation of uncertainty
The methods for assessing uncertainty in a probabilistic framework can be divided into two categories; firstorder (or second-order) reliability methods, referred to as FORM (or SORM), and Monte Carlo simulation methods.6
Both FORM and SORM are used for examining the reliability of the system, which could be expressed as the
failure probability, i.e. the probability of an event. The methods use transformations and approximations evaluating
the surroundings of a design point in order to examine the probability that potential uncertainties cause a failure.
A Monte Carlo simulation draws samples from the probability distributions of the uncertainty sources and
calculates the values of the system characteristics for each sample. The resulting probability distributions for the
system characteristics will finally converge to the true probability distributions according to the law of large
numbers, but large may mean thousands of samples.4,12
More efficient sampling methods have been developed, divided into optimal designs and space-filling designs.13
Optimal designs focuses on optimizing specific criterions, such as prediction variance, and include amongst others
the I-optimal and D-optimal designs.13 Space-filling designs aim at spreading the samples over the design space and
includes designs such as sphere packing design, Latin hypercube sampling (LHS) and the maximum entropy
design.8
Latin Hypercube Sampling divides the design space into equally probable intervals according to the probability
distributions of the parameters.14 Then it draws one sample from each interval, and it has been demonstrated to give
accurate estimations of the mean and variance of the system characteristics for just tens of samples.12,13 In Ref. 12,
LHS was compared to Monte Carlo simulation, based on both the true model and a meta-model, for a simulation
based aircraft design example. The conclusion was that LHS with 50 samples gives approximately the same mean
value and standard deviation as a Monte Carlo simulation of 20 000 samples. Unfortunately, the actual probability
distributions vary slightly.
2
American Institute of Aeronautics and Astronautics
III. The modified optimization algorithm
There exist many optimization algorithms, ranging from gradient-based methods via direct search methods to
evolutionary algorithms.15 As base for the modified algorithm and for the comparison in this paper the Complex-RF
method as described in Ref. 9 is used. The performance of the Complex-RF algorithm compared to other
optimization algorithms is demonstrated in Ref. 15.
A. The Complex-RF optimization algorithm
The Complex-RF algorithm uses k number of points, where k typically is around twice the number of
optimization variables, n. The figure in Rn with k points is referred to as a Complex. The algorithm starts by placing
k points randomly inside the design space. The objective function values are then compared and the point with the
worst function value is reflected through the center of the remaining points. If the function value of the new point
still is worse than the other function values, the point is moved halfway towards the center of the other points. This
process continues until the new point stops being worst. Now a new point is worst and reflected through the centroid
of the complex. This process is continued until convergence or the maximum number of iterations is reached.
B. Modified Complex-RF used in this work
A commonly used approach to reduce the time it takes to conduct an optimization is to create and fit a global
meta-model before the optimization is started and using the meta-model in the optimization algorithm.16 When the
optimum value of the meta-model has been determined, the original model is simulated once with the optimal
parameters to reveal the corresponding objective function value. The drawback with this approach is that it requires
many samples to fit a global meta-model accurately.8
Start
Optimization
Create Initial Points
Evaluate Points
True Model
Enough
Points?
Yes
Create MM
No
Stop Criteria
Fulfilled?
End
Optimization
Yes
No
MM Check
Reflect worst point
through centroid
True Model
/ MM
Evaluate point
Evaluate point
No
True Model
/ MM
Move point inwards
Is the point
still worst?
Yes
Figure 1. A basic overview of the modified Complex algorithm
Instead of creating a meta-model before the optimization, a modified version of the Complex-RF algorithm
(Complex-RFM) is developed where the meta-model evolves as the optimization progresses. A basic flowchart of
the algorithm can be seen in Figure 1. The boxes with grey background are the ones which have been added or
modified from the original Complex algorithm. The box named “Enough Points?” represents an activity which
calculates if the initial points are enough to create an initial meta-model, and if the answer is positive a meta-model
is created. The boxes named “True Model / MM” have replaced calls of the objective function and follow the
workflow seen in Textbox 1. In short, it means that the true model is called until there exist enough samples to
3
American Institute of Aeronautics and Astronautics
create a meta-model and for later iterations the meta-model is called instead of the true function. Hence, the
computational burden is significantly reduced.
Textbox 1. The Pseudo coding for the “True Model / MM” box in Figure 1.
Is there an existing MM?
Yes:
Call the MM to receive the value of the objective function
No:
Call the true model to receive the value of the objective function
Has the true model been called enough times to create a MM?
Yes:
Create a MM
One of the difficulties with creating and using meta-models during the optimization is to decide when to call the
original model and when to call the meta-model. This is handled by the box named “MM Check”, which is called
after each iteration of the Complex algorithm, and it follows the pseudo code in Textbox 2. If the two latest metamodels are deemed equal it is likely that the meta-model accurately reflects the behavior of the objective function in
the vicinity of the last created point. For simple functions this may indicate that the latest meta-model can be used
for the rest of the iterations. However, since the created meta-models are local, they might be inaccurate for other
parts of the design space. Therefore, the optimization algorithm just skip calling the true model for a certain number
of iterations, see the pseudo code below.
Textbox 2. The Pseudo coding for the “MM Check” box in Figure 1.
Was the function value of the last point calculated from the true model?
No:
Are the two latest MMs equal?
Yes:
Skip "Model Check" for p number of iterations
No:
Call the true model to get the true value for the moved point
Update the MM
The local meta-models created and used by the modified optimization algorithm is a second order response
surface on the form seen in Eq. (1).8 This enables the comparison of the two latest meta-models to be performed by
comparing the coefficients, βi.
∧
y = β0 + β1 x1 + β 2 x2 + β11 x12 + β 22 x 22 + β12 x1 x2
(1)
A second order response surface of the form seen in (1) is not suitable as a global meta-model of a multimodal
objective function since it only approximates the main trends of the function.8 However, since the meta-models are
created and fitted during the optimization process, they will be fitted locally and consequently often have good
accuracy where they are intended to be used.
Another drawback with using a second order response surface as a meta-model inside the optimization is that the
required number of samples to fit the response surface approximately increases with the square of the number of
variables. This makes this algorithm inefficient for solving optimization problems with many variables since the
meta-models cannot be created until later iterations. Therefore it is recommended to lower the number of variables
before starting the optimization, for example by performing a sensitivity analysis.
The optimization algorithm may often end with calling the meta-model the last iterations since the points are
close to each other then. Consequently, the most interesting information received from the optimization is the
optimal parameter values and similarly to the approach with an optimization performed on a global meta-model, the
original model should be called with the optimal parameters to receive the true model results.16
Additionally, the properties of the identified optimum could be assessed by studying the identified coefficients of
the latest created meta-model. In a real world problem where the objective function is based on simulation of nonlinear models, the shape of the objective function is unknown. Therefore the meta-model will not only speed up the
optimization, it will also give great insight into the properties of the problem as well. For example, the robustness of
the optimum could be assessed by studying the coefficients of the response surface equation. When analyzing the
result, it could be seen which design parameters that contribute most to the objective function.
4
American Institute of Aeronautics and Astronautics
IV. Demonstrations of the modified Complex-RF algorithm
To determine whether the proposed algorithm is applicable for engineering problems, it is tested for a few
analytical functions and an industrial example. It is also compared with the original Complex-RF algorithm to
estimate its efficiency.
A. Comparison of hit-rate and number of calls
To ensure that the percentage of optimizations that converge to the global optimum, the hit-rate, of the complex
algorithm is not lowered dramatically by creating and using meta-models during the optimization, the modified
algorithm is compared with the original one for two analytical functions. The two functions are a simple objective
function of the form seen in Eq. (2), whereas the second is the somewhat more complicated peaks-function which
may be called in MATLAB by writing “peaks” in the command window, see Eq. (3).
min f ( x1 , x2 ) = ( x1 − 1) + ( x2 − 2 )
2
s.t.
2
− 3 ≤ xi ≤ 3 , (i = 1,2 )
{
(2)
}
{
x

2
2
min f (x ) = 3(1 − x1 ) exp − x12 − ( x2 + 1) − 10 1 − x13 − x2  exp − x12 − x22
5

{
1
2
− exp − ( x1 + 1) − x 2
3
s.t.
}
}
− 3 ≤ xi ≤ 3 , (i = 1,2 )
(3)
In order to evaluate the performance of the Complex-RFM algorithm, the number of function calls to the real
(computational expensive) model will be compared for the two algorithms. The median number of real model calls,
number of points evaluated and number of successful optimizations out of 1000 for problem (2) and (3) can be seen
in Table 1.
Table 1. Hit-rate and number of function calls for equations (2) and (3) respectively
Eq. (2)
Peaks
Complex-RF Complex-RFM Complex-RF Complex-RFM
Hitrate [%]
99.9
99.4
71.9
56.5
Number of points evaluated
148
148
113
231
Number of real model calls
148
15
113
45
As can be seen, the meta-model approach is efficient for both functions but excels for the simple test function
when the second order response surface can match it perfectly. For the Peaks function, the hit-rate is lowered by
several percentages, but the number of real model calls is more than halved. Consequently, it is possible to perform
two optimizations with the meta-model approach faster than one optimization with the original algorithm. The
probability of finding the optimum with two optimizations using the modified algorithm is higher than finding the
optimum by performing one optimization with the original one. Therefore, it is possible to claim that Complex-RFM
is efficient for more computationally expensive models.
The hit-rate of the Complex-RF algorithm may seem a bit low, but in Ref. 15, the Complex-RF algorithm was
compared against the gradient-based algorithm fmincon in MATLAB (i.e. SQP) and a genetic algorithm with
respect to hit-rate and number of function evaluations for the peaks function. This comparison is presented in Table
2, and as can be seen the Complex-RF algorithm displays an adequate tradeoff between hit-rate and number of
function evaluations.
5
American Institute of Aeronautics and Astronautics
Table 2. Hitrate and number of function calls for three different optimization algorithms
Complex-RF Genetic Algorithm Fmincon
Hitrate [%]
73
99
28
Number of evaluations
94
442
35
An example of an obtained response surface, which was created during an optimization of the peaks-function
with the Complex-RFM algorithm, can be seen in Figure 2. The response surface gives an excellent approximation
of the behavior of the objective function around the optimum, and it is obtained without any extra calls to the
computational expensive model.
Figure 2. The last response surface plotted in the same graph as equation (3).
a) Contour plot
b) 3D-figure
Figure 3. Contour plot and 3D-figure for equation (4).
B. Demonstration of reliability based design optimization capabilities
It is also of interest to evaluate if the modified algorithm can be used for RDO or RBDO. This is tested by
performing a RBDO for the analytical function seen in Eq. (4) and visualized in Figure 3a) and b), respectively.
min f ( x1 , x2 ) = g ( x1 , x2 ) =
1 m
∑ g (x1 j , x2 j )
m j =1
6
American Institute of Aeronautics and Astronautics
  x1 − 0.2  2  x2 − 0.2  2 
  x1 − 0.6  2  x2 − 0.6  2 
g ( x1 , x2 ) = 2 − exp− 
 −
  − exp− 
 −
 
  0.1   0.1  
  0.4   0.4  
s.t.
0 ≤ xi ≤ 1 , (i = 1,2 )
(4)
To create probability distributions for the variables, both x1 and x2 are considered to follow uniform distributions
with bounds ±10% around the current value. The probability distributions for the function value g(x1,x2) are created
by LHS consisting of 50 samples.
As objective function for the optimization, the mean of g(x1,x2) is used since it is an estimation of the expected
value of the function. In a real world application, g might be an entity such as the maximum stress within a product
and the interpretation would be that the expected stress in the product is minimized.
Table 3. Results from a probabilistic optimization of equation (4)
Optimal point
Number of points evaluated Number of model calls
x=[0.6009, 0.6075]
490
200
Number of samplings
10000
The results from the optimization are presented in Table 3 and the resulting point is plotted in Figure 3 together
with the contour plot of the function. It can be noted that the point is not lying in the global optimum of the function
g(x1,x2) and the explanation is that the function values in the vicinity of the global optimum (x=[0.2 0.2]) are higher
than those around the local optimum (x = [0.6 0.6]). The optimal point from a reliability point of view is the local
optimum and according to Figure 3, the algorithm has found the correct solution.
C. Demonstration for an aircraft application
To further improve the confidence in the modified algorithm, it is used for optimizing an industrial application.
The application is a system model of a dynamic pressure regulator found in airplanes and modeled in Dymola. The
purpose of the dynamic pressure regulator is to control the air pressure delivered to the environmental control
system. Consequently, the most important properties of the dynamic pressure regulator is the time it takes to fill the
environmental system, tf, and the end pressure inside the environmental control system when it has been filled, pe. A
screenshot of the Dymola model is presented in Figure 4 and a more thorough description of the dynamic pressure
regulator can be found in Ref. 12.
Figure 4. Screenshots from Dymola displaying the pressure regulator and the
components inside it.
7
American Institute of Aeronautics and Astronautics
A sensitivity analysis is performed to identify the most important system parameters and the result can be seen in
Table 4. The three most important system parameters are the pressure from the tank, pt, the maximum opening area
of the vent valve, Av, and the mass of the piston inside the main valve, mp. Therefore they are chosen as variables
for the optimization. They are considered to follow uniform distributions with upper and lower bounds lying five
percent to the sides of the normative values.
Max opening area of
on/off-valve
Max opening area of
vent valve, Av
Max opening area of
main valve, Am
Friction coefficient
in main valve
Mass of piston, mp
Preloading of piston
Piston area for
output flow, A2
Piston area for
support pressure As
pe
tf
Tank pressure, pt
Table 4. Normalized sensitivities for the dynamic pressure regulator
1.1
1.4E-4
0.13
-5.7E-5
-1.0
-8.4E-10
0.35
2.5E-4
0.055
0.0075
0.053
0.50
0.033
4.6E-4
0.15
-0.25
0.013
-0.26
Since the end pressure and filling time are of importance to the functionality of the dynamic pressure regulator,
the chosen optimization problem is a combination of both. The aim is to minimize the filling time while the end
pressure with a 90 % probability is lying between an upper and lower boundary. As objective function, a LHS
consisting of 100 samples is used to estimate the expected value of the filling time as well as the probability
distribution of the pressures.
The optimal parameter values obtained from the RBDO with the modified algorithm can be seen in Table 5,
together with the corresponding values received from a deterministic optimization with the Complex-RF algorithm.
Table 5. Optimal parameter values obtained from the two optimizations
Deterministic optimization
RBDO
Variable
Unit
with Complex-RF
with the Complex-RFM
pt
Tank pressure
[Pa]
492 000
499 000
Av Max opening area of vent valve [m2]
9.74E-5
3.87E-4
mp
Mass of piston in main valve
[kg]
2.42E-4
2.44E-4
Number of evaluations
868
1000
Number of LHS performed
57
As can be seen, the optimum parameter values for the two approaches differ slightly, which indicates that there
is an alternative design which is more robust compared to the one obtained using the deterministic approach.
Furthermore, conducting a sensitivity analysis around the two obtained optima supports the statement that the
optimum obtained from the RBDO with the Complex-RFM algorithm is more robust. This is demonstrated in Table
6, where normalized sensitivities for the two optima are presented. The significantly lower number of LHS
performed than the number of evaluations for the RBDO with Complex-RFM depends on the syntax of the
algorithm, where a LHS only is performed after the latest moved point is no longer worst. Consequently, the metamodel is called for estimating the objective function value when a point is moved. For industrial applications, such
as the one presented in this paper, it is not unusual that the point may need to be moved many times before it is no
longer worst.
Table 6. Normalized sensitivities at the two optima.
Tank pressure, pt
Deterministic Optimization
with Complex-RF
RBDO with Complex-RFM
pe
tf
pe
tf
0.0129
-0.123
4.10E-4
0.0475
Max opening area
of vent valve, Av
8.32E-3
-0.0264
-9.67E-4
0.0488
8
American Institute of Aeronautics and Astronautics
Mass of piston, mp
3.01E-3
0.0941
-4.50E-4
0.0352
D. Further comments
It is possible to make RDO and RBDO even more computationally efficient by using lower layers of metamodels as seen in Figure 5. Figure 5a) displays the approach that is used in section IV-C, where the only created
meta-models are the ones created by the modified optimization algorithm. The true model is called many times since
the sampling process of 100 samples is performed for each iteration. These calls are limited in the approach in 5b),
where a meta-model of the original model is built and the sampler calls the meta-model instead of the original
model. Approach 5c) focuses on creating meta-models of the mean and variance of the interesting characteristics of
the true model as functions of the model parameters. The last approach, 5d), is a combination of a), b) and c), where
meta-models are created for the model, the mean and variance, and the objective function values. It is probably
impractical to use d), because it uses three layers of meta-models since each meta-model is an approximation and
consequently more errors are introduced for each layer of meta-models.
A relatively large number of simulations of the model need to be performed in c) to fit the meta-model since the
100 simulations performed with LHS only represent one point for the meta-model. This number may be of about the
same order of magnitude as the required number of objective function calls for the Complex-RFM algorithm,
meaning that the meta-model will barely be used before the optimization is complete. Approach b), creating a metamodel of the model itself, is probably better since 100 simulations of the model are performed during each iteration
of the optimization process. The 100 simulations per initial point in the optimization algorithm will most likely be
enough to yield accurate results if a global meta-model well suited for covering the design space is used. This
approach is preferable for highly expensive computer models, but is not applied in this work since the model of the
dynamic pressure regulator only requires one second of simulation time.
a) Using the modified optimization algorithm
b) A MM of the true model + a)
c) A MM of the mean/variance of the process + a)
d) A combination of a), b) and c)
Figure 5. Uses of MMs on different levels of RBDO
V. Conclusion
In this paper, a modified version of the Complex algorithm is presented, which automatically creates metamodels as the optimization progresses. Optimization is then conducted based on the meta-model instead of the
computationally more expensive real model. The algorithm has been applied to deterministic as well as stochastic
optimization problems based on both analytical functions and a real industrial example. The algorithm shows great
potential since the usage of meta-models reduces the computational time as the number of calls to computationally
expensive models could be decreased. Moreover, the meta-models are generated automatically, so there is no extra
burden on the user of the method.
By studying the meta-model itself, great insight to the properties of the problem could be gained. In the general
case, where the objective function is based on non-linear simulation models, the shape of the objective function is
not known. However, by studying the coefficients of the meta-model, the properties of the problem could be
assessed. Furthermore, by calculating the derivatives of the meta-model with respect to the design variables, the
sensitivities could be obtained. It is also possible to build meta-models of other properties than just the objective
function, such as sub-objectives or constraints, to give even more insight to the problem at hand.
A drawback with the algorithm is that it is inefficient for problems with many variables since the required
number of samples to fit a second order response surface increase with the number of variables squared. However,
other kind of meta-models (e.g kriging or neural networks) could be used as well. On the other hand, a response
surface has the benefit that it reveals the behavior of the objective function around the optimum point in an easy to
grasp manner. Furthermore, a quadratic response surface should be able to obtain good accuracy as the meta-model
only represent the true model locally at an optimal point where a quadratic performance could be anticipated.
Future work includes evaluation of other meta-modeling techniques, as well as applying the method to more
robust design optimization and reliability based design optimization problems. It would also be interesting to study
the obtained meta-model in more detail in order to gain further insight into the properties of real world problems.
9
American Institute of Aeronautics and Astronautics
Furthermore, the general idea of creating meta-models of computational expensive models as the optimization
evolves could be applied to other optimization algorithms as well. The technique should be best suited for other
population based methods such as genetic algorithms or particle swarms.
Acknowledgments
The research performed in this paper has received founding by the European Community’s Seventh Framework
Program under grant agreement no. 234344 (www.crescendo-fp7.eu).
References
1
Mavris, D.N., DeLaurentis, D.A., Bandte, O. and Hale, M.A.,”A Stochastic Approach to Multi-Disciplinary Aircraft
Analysis and Design”, AIAA 98-0912, 1998.
2
Beckwith, T. G., Marangoni, R.D. and Lienhard V, J.H., Mecahnical Measurements, sixth edition, Pearson Prentice Hall,
Upper Saddle River, NJ, 2009, Chaps. 3.
3
Andersson J., “Multiobjective Optimization in Engineering Design - Applications to Fluid Power Systems”, Dissertation,
Linköping studies in Science and Technology, Dissertation No. 675, Linköping University, Linköping, Sweden, 2001.
4
Lönn, D., “Robust design - Accounting for uncertainties in engineering”, Linköping Studies in Science and Technology.,
Thesis No 1389, Linköping University, Linköping, Sweden 2008
5
Padulo, M., “Computational Engineering Design under uncertainty – An aircraft conceptual design perspective”, Cranfield
University, PhD Thesis, July 2009.
6
Martin, J. D. and Simpson, T. W., “A methodology to manage uncertainty during system-level conceptual design”,
Proceedings of IDETC/CIE 2005 ASME 2005 International Design Engineering Technical Conferences & Computers and
information in Engineering Conference, 2005, pp. 1-11.
7
Martin, J. D. and Simpson, T. W., “A Monte Carlo Simulation of the Kriging Model”, 10th AIAA/ISSMO Symposium on
Multidiscipilinary Analysis and Optimization, AIAA-004-4483, 2004.
8
Myers, R.H., Montgomery, D.C. and Anderson-Cook, C.M., Response Surface Methodology, Process and Product
Optimization Using Designed Experiments, Third Version, John Wiley & Sons, Inc., Hoboken, New Jersey, 2009.
9
Krus, P. and Andersson, J., "Optimizing Optimization for Design Optimization", Proceedings of ASME Design Automation
Conference , Chicago, USA, September 2-6, 2003.
10
Saltelli, A., Chan, K. and Scott, E.M., Sensitivity Analysis, John Wiley & Sons, Chichester, West Sussex, 2000, pp. 10.
11
Steinkellner, S. and Krus, P., "Balancing Uncertainties in Aircraft System Simulation Models", 2nd European Air & Space
Conference CEAS, Manchester, United Kingdom, 2009.
12
Persson, J. A., and Ölvander, J. “Comparison of Sampling Methods for a Dynamic Pressure Regulator, AIAA-2011-1205,”
Proceedings of the 49th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition,
AIAA, Orlando, FA, 4-7 Jan. 2011.
13
Johnson, R.T., Montgomery, D.C., Jones, B., and Fowler, J.W., “Comparing Designs for Computer Simulation
Experiments”, Proceedings of the 2008 Winter Simulation Conference, 2008.
14
McKay, M.D., Beckman, R.J. and Conover, W.J., “A Comparison of Three Methods for Selecting Values of Input
Variables in the Analysis of Output from a Computer Code”, Technometrics, Vol. 21. No. 2 , 1979, pp. 239-245.
15
Ölvander J., and Krus P., "Optimizing the Optimization - A Method for Comparison of Optimization Algorithms",
proceedings of the 2nd AIAA Multidisciplinary Design Optimization Specialists Conference, Newport, RI, USA, May 1-4, 2006.
16
Duvigneau, R., and Praveen, C., “Meta-Modeling for Robust Design and Multi-Level Optimization”, 42nd AAAF Congress
on Applied Aerodynamics, Sophia-Antipolis, France, March 2007
10
American Institute of Aeronautics and Astronautics