Quantitative methods in finance time series analysis

《Quantitative methods in finance time series analysis》由会员上传分享，免费在线阅读，更多相关内容在学术论文-天天文库。

QuantitativeMethodsinFinance:TimeSeriesAnalysisMarcS.PaolellaMarch2005 ParameterEstimationandLikelihoodDataisnotinformation,informationisnotknowledge,knowledgeisnotunderstanding,understandingisnotwis-dom.(Cli®StollandGarySchubert)Donotputfaithinwhatstatisticssayuntilyouhavecare-fullyconsideredwhattheydonotsay.(WilliamW.Watt)Commonsense"isnotcommonbutneedstolearntsys-tematically....Asimpleanalysis"canbeharderthanitlooks....Allstatisticaltechniques,howeversophisticated,shouldbesubordinatetosubjectivejudgement.(Chat¯eld,1985,p.228)1 2Question:Interestcentersonhowlongittakesawomantobecomepregnantusingaparticularmethodofassis-tance,e.g.,temperaturemeasurements,hormonetreat-ment,etc.Imagineastudyinwhicheachofryoungcouples(independentlyfromeachother)attemptstocon-ceiveeachmonthuntiltheysucceed".Lettherequirednumberofmonthsforeachcouplebede-notedbyXi,i=1;:::;r.GivenvaluesX1=x1;:::;Xr=xr,howwouldyoucomputePr(X>5),i.e.,theproba-bilitythatittakesmorethan5monthstoconceive?AnswerI:Ingeneral,giventheindependentandidenticallydis-tributed(iid)datasetX1;:::;Xn,considertheratherintuitiveestimatorofPr(X·t):Xn¡1pbt=Pr(X·t)=nI(¡1;t)(Xi)i=1whereI(¡1;t)(Xi)istheindicatorfunction,de¯nedforeverysetMµRas(1;ifx2M;IM(x)=0;ifx=2M:Thisiscalledtheempiricalcumulativedistributionfunc-tion,abbreviatedtheempiricalcdf,orjusttheecdf. 3Let'ssimulateasetof15couplesusingMatlab:p=0.3;%probabilityofsuccessinanygivenmonthn=15;%howmanycouplesdata=geornd(0.3,15,1)+1%geometricrandomvariables!Herearesometypicalvalues:6131542438123243Nowweneedtocomputeandplottheempiricalcdf.Firsttabulatetheresultswitht=tabulate(data),giving1213.332320.003320.004320.00516.66616.6670816.66901001101201316.66 4Nowlet'scomputetheecdfandplotit:ecdf=cumsum(t(:,2));grd=t(:,1);plot(grd,ecdf/n,'b-o'),set(gca,'fontsize',14)h=line([55],[01]);set(h,'linestyle','--')h=line([05],[0.80.8]);set(h,'linestyle',':','linewidth',2)10.90.80.70.60.50.40.30.20.1002468101214Figure1:Empiricalcdf²Nowwecananswerouroriginalquestion:Fromtheecdf,weseethatanestimateofPr(X>5)is1¡0:8=0:2.²Therearesomedrawbackswiththismethod.OneisthattheprobabilityofX>13iszero,whichissilly.Itisanaresultofthefactthatwehavealimitedamountofdata.²Anotherproblemisthatourintuitionwouldsuggestthattheecdfcurveshouldbesmoothintheory. 5ParametricModelForourexample,whatistheprobabilitymassfunction(pmf)cor-respondingtoacouple?Inparticular,letXbethenumberoftrialswhichareobserveduntil(andincluding)the¯rstsuccess"occurs.WhatisPr(X=x)?Letpbetheprobabilityofsuccessinanymonth.ThenPr(X=1)=pPr(X=2)=p(1¡p)and,ingeneral,x¡1Pr(X=x)=p(1¡p)x=1;2;:::or,withourindicatorfunctionnotation,x¡1Pr(X=x)=p(1¡p)If1;2;:::g(x):IfXhasthispmf,thenXiscalledageometricrandomvariable(r.v.)withsuccessprobabilityp.Whataboutthecdf?XxFX(x;p)=fX(k)(de¯nitionofcdf)k=1Xxk¡1=p(1¡p)(putinthepmf)k=1x=1¡(1¡p)(algebra:homework!)forx=1;2;:::,andzerootherwise. 6Likelihood{BasedInferenceLikelihoodisthesinglemostimportantconceptofstatis-tics.(OleE.Barndor®{Nielsen,1991,p.232)Thelikelihoodfunction,L,isthejointdensityofasampleX=(X1;:::;Xn)viewedasafunctionoftheunknownscalarparameterµconditionalontheobservedsamplevaluesX=x,i.e.,L=L(µjx)=fX(xjµ).Inourexample,theunknownparameterisp.Youcancallitµ!Theentirelikelihoodfunctionisofinterest,andcanbeusedtoanswermanyquestions.However,greatattentionispaidtothesinglevalueofµwhichmax-imizesthelikelihoodfunctionforagivendataset:Itisreferredtoasthemaximumlikelihoodestimator,abbreviatedMLE,anddenotedµ^(or,todistinguishitfromotherestimators,µ^ML). 7OfgreatuseisthenaturallogarithmofL,whichwewillabbreviateas`=`(µjx)=lnL.Forfurthernotationalconvenience,let`_denotethe¯rstderivativeof`withrespecttoµ(asopposedtotheubiquitoususageoftime).Similarly,thesecondderivativeof`isdenoted`Ä.Thescorefunctionofasampleisde¯nedtobe`_.AfundamentalresultisthatE[`_]=0,whichisshownasfollows.FirstnotethatfX(xjµisafunctionofbothxandµ(sothat,forexample,itmakessensetointegratew.r.t.x,andtodi®erentiatew.r.t.µ).Inthefollowingproof,wejustwritef(x)soitiseasiertoread.·µ¶¸dE[`_]=Elnf(x)dµZµ¶d=lnf(x)f(x)dxdµZµ¶1d=f(x)f(x)dxf(x)dµZd=f(x)dxdµd=1dµ=0;Theexchangeofderivativeandintegralcomesfromaregularityconditionwhichweassumeholds. 8iidPnAssumeX1;:::;Xn»Geo(µ),andde¯nes=i=1xi.ThejointdensitycanbewrittenYnfX(x;µ)=fXi(xi;µ)i=1Ynxi¡1=µ(1¡µ)If1;2;:::g(xi)i=1Ynµxi=(1¡µ)If1;2;:::g(xi)1¡µi=1µ¶nYnµs=(1¡µ)If1;2;:::g(xi)1¡µi=1The(natural)logisXn`=nlnµ¡nln(1¡µ)+sln(1¡µ)+lnIf1;2;:::g(xi):i=1Its¯rstderivativew.r.t.µisnnsnn¡s`_=+¡=+µ1¡µ1¡µµ1¡µandsettingthistozeroyieldsn1µ^ML==;sX¹whereX¹isthesampleaverageofthedata.Forourdatasetwith15observations,^pML=0:246. 9InvarianceTheMLEpossessesaveryimportantproperty,whichisreferredtoasinvariance:TheMLEofsomefunctionofparameterµ,sayg(µ),isgivenbyg(µ^ML).ThispropertyisNOTtypicalformanyestimators,andisaveryusefulaspectofmaximumlikelihood.Forexample,theoddsofnotgettingpregnantis(1¡p)=p.ItsMLEis(1¡p^ML)=p^ML¼3:Recallthecdfofthegeometricdistribution:xFX(x;p)=1¡(1¡p)forx=1;2;:::,andzerootherwise.Accordingtotheinvarianceprinciple,theMLEofPr(X>5)=1¡Pr(X·5)isjust51¡FX(5;^pML)=(1¡p^ML)=0:244;whichissimilar"toournonparametricestimateof0.200. 10Wecanoverlaytheecdfwiththeparametricallyestimatedcdfandseethedi®erence:holdon%overlaygraphicsdatagrd=0.1:0.1:15;%gridofpointsfortheMLEmle=1-(1-phat).^grd;%computetheMLEh=plot(grd,mle,'r--');%plotitasared,dashedlineset(h,'linewidth',2)%makeitabitthickerholdoff10.90.80.70.60.50.40.30.20.10051015Figure2:Empiricalcdfandparametricallyestimatedcdf 11ComparisonofEstimationStrategiesMostreallifestatisticalproblemshaveoneormorenon-standardfeatures.Therearenoroutinestatisticalques-tion;onlyquestionablestatisticalroutines.(D.R.Cox,quotedinChat¯eld,1991)Question:Shouldtheempiricalcdforaparametricmethod(i.e.,theMLE)beusedforestimatingPr(X·t)?Answer:Itdepends.Theanswerdependsontheextenttowhichtheparametricformofthedataisknown.Review:Considerourpregnancyexample,andassumethatin-ferenceonfertilityisbasedonastudyconductedon15couples.Ifawomanisinterestedinknowingtheprobabilitythatshebe-comespregnantwithinthenextfourmonths,thentherearetwopossibleestimators:Oneisthenonparametriconebasedontheempiricalcdf,whichsimplycomputesthefractionofthe15coupleswhoconceivedwithinfourmonths.Theparametricmethodinvolves¯rstestimatingtheprobabilitypassociatedwiththegeometricmodel,thencomputingthecdfofageometricr.v.withprobability^p,forfourmonths. 12Whatdowemeanby:Themodeliscorrect"?Thismeans:Theprobabilityofofgettingpregnant²foraparticularwomanisconstantforeachmonth,²isthesameforallwomeninthestudy²isalsothesameinthegeneralpopulationwishingtousethisinformationtodeterminetheirownchances.Aretheseassumptionsrealistic?Almostcertainlynot!IftheyAREtrue,then,generallyspeaking,theparametricmethodwillleadtomoreaccurateanswers.Why?Because,intheestimationofp(and,hence,theestimationofthegeometriccdf),thewholedataisusedinanoptimalway.Inthenonparametricmethod,anobservationiseither·4,oris>4,i.e.,thedatapointsarebeingreducedtoBernoullirandomvariablesandinformationisbeinglost.Ontheotherhand,iftheassumedparametricmodeliswrong,theninferencebasedonitcouldbemisleading. 13Imaginethattwoindependentpregnancystudies,AandB,eachwith100couples,willbeconducted.Eachusesadi®erentmethodofassistanceinconceiving.Ahistogramoftheresulting100measurements(numberofmonthsuntilsuccess)forthe¯rststudyisshownintheleftupperpanelofFigure3.3010.9250.8200.7150.60.5100.450.300.2024681012140246810121416401350.9300.8250.7200.6150.5100.450.300.2051015202530350510152025Figure3:Comparisonofbehaviorofcorrectlyspeci¯edandmisspeci¯ed¯ttedcdfs. 14Therightupperpanel,solidline,showstheempiricalcdfoverlaidwiththeparametriccdfofageometricr.v.(dashedline)usingtheestimatedvalueofp,whichwas0:31.Itcertainlyappearsfromthecdfplotthattheempiricaland¯ttedcdfsareverycloseandwoulddeliverpracticallythesameinforma-tion.ThedataforstudyAwere,infact,100simulatedrealizationsofageometricr.v.withsuccessprobability0:3.ThelowerpanelsofFigure3aresimilartotheupperones,butcorrespondtostudyB,forwhich^p3=0:27.Althoughthetwoparameterestimatesarerelativelyclose,thehistogramofthedataforstudyBdi®ersremarkablyfromthatofstudyA.Moreover,thereisalargediscrepancybetweentheempiricaland¯ttedcdf.Thereasonisthatthegeometricmodelforthedataiswrong.ForstudyB,eachofthe100datapointsweregeneratedasfollows:Withprobability1/3,itisarealizationofageometricr.v.withp=0:1;withprobability2/3itisgeometricwithp=0:5.ThefollowingMatlabcodewasused:n=100;p1=0.1;p2=0.5;y=zeros(n,1);fori=1:nifrand<0.3333,y(i)=geornd(p1,1,1);else,y(i)=geornd(p2,1,1);endend,y=y+1; 15Theresultingdatasetisamixturemodelandisnotterriblyunrealistic!Imagine(somewhatsimplisticallyofcourse)thatcoupleshavingtroubleconceivinghaveaprobabilityof0.1ofsuccess.Thetreat-mentusedinstudyBbooststheprobabilityto0.5,butitonlyworksin2/3ofwomen.Thisgivesrisetothemixture.Infact,amixtureofgeometricdistributionsisgenerallyconsideredtobeagooddescriptionofactualtimeuntilconception;see,forexample,EcochardandClayton(2000)andthereferencestherein.ObservehowinthecdfplotsforstudyB,the¯ttedcdf¯rstunderes-timatesthenoverestimatestheempiricallyobservedprobabilities,i.e.,itisattemptingtocompensateasbestitcan,sothatnosinglediscrepancyistoogreat.Themaximaldiscrepancyserves,infact,asanaturalmeasureofthegoodnessof¯tofaparametricmodelandisreferredtoastheKolmogorovdistance,givenby¯¯¯¯KD=100£sup¯Fbemp(x)¡FbPara(x)¯;x2RwhereFbempdenotestheempiricalandFbParathe¯tted(orparamet-ric)cdf.ThisstatisticisdiscussedfurtherinDeGroot(1986)andD'AgostinoandStephens(1986). 16FurtherExamplesiidExample(Poisson)LetXi»Pois(µ),i=1;:::n.RecallthatthepmfofthePoissonise¡µµxfPois(xi;µ)=Pr(X=x;µ)=;x=0;1;::::x!PnWithX=(X1;:::;Xn)ands=i=1xi,YnYne¡µµxie¡nµµsL(µjX)=fPois(xi;µ)==Qni=1i=1xi!i=1xi!and`=¡nµ+slnµ¡ln(x1!¢¢¢xn!);`_=¡n+s=µ;fromwhichitfollowsthatµ^ML=S=n=X¹.Also,`Ä=¡sµ¡2;sothatX¹isindeedamaximum.¥iid¡¢Example(normal)LetX»N¹;¾2,i=1;:::n,eachwithidensity(µ¶)211x¡¹fX(x;¹;¾)=pexp¡:2¼¾2¾Q¡¢n2Assume¹isunknown,but¾known.ThenL(¹jX)=i=1Áxi;¹;¾andnn¡¢1Xn22`(¹)=¡ln(2¼)¡ln¾¡(xi¡¹);222¾2i=1Pwith`_=¾¡2n(x¡¹).Settingtozeroyields^¹=X¹.¥i=1iML 17ExampleRecallthattheexponentialdistributioncanbeusefulinmodelingthetimeuntilacomponent(say,anelectricaldevice)iidfails.So,letXi»Exp(¸),i=1;:::;nbeasampleofsuchtimes,butassumethatonlythe¯rstkfailures,k·n,areobserved,i.e.,onlythe¯rstkorderstatistics,Y=(Y1;:::;Yk)areavailable.Itcanbeshownthattheirjointdensityisn!Ykn¡kL(¸jY)=[1¡F(yk)]f(yi)(n¡k)!i=1Ã!n!Xkk=exp(¡¸(n¡k)yk)¸exp¡¸yi(n¡k)!i=1Xkk/¸exp(¡¸T);T=yi+(n¡k)yki=1and,settingthe¯rstderivativeofLor`tozeroandsolving,¸^ML=k=T.²Ifk=n,then¸^MLsimpli¯esto1=X¹n.²Noticealso:Ifk=n,then¸^ML=1=X¹nistheSAMEastheMLEinthegeometricmodel.BOTHmodels(canbeusedto)measuretimeuntilsomethinghappens",withthegeometricbeingadiscreter.v.,andtheexponentialbeingacontinuousone.²Iftheexponentialdistributionwereparameterizedas¡1fX(x;µ)=µexp(¡x=µ)instead,thentheinvariancepropertyoftheMLEimpliesimmediatelythatµ^ML=1=¸^ML=T=k(thussavingyoulotsofwork!)¥ 18 ImportantDistributionsandTransformationsItisremarkablethatasciencewhichbeganwiththecon-siderationofgamesofchanceshouldhavebecomethemostimportantobjectofhumanknowledge.(PierreSimonLaplace)Lifeisgoodforonlytwothings,discoveringmathematicsandteachingmathematics.(Sim¶eonPoisson)Iadvisemystudentstolistencarefullythemomenttheydecidetotakenomoremathematicscourses.Theymightbeabletohearthesoundofclosingdoors.(JamesCaballero)Alectureisaprocessbywhichthenotesoftheprofessorbecomethenotesofthestudentswithoutpassingthroughthemindsofeither.(R.K.Rathbun)19 20RandomVariableTransformationIfXisacontinuousrandomvariablewithpdffXandgisacon-tinuousfunction1withdg=dx6=0(overthesupportofX),thenfY,thepdfofY=g(X),canbecalculatedby¯¯¯dx¯fY(y)=fX(x)¯¯¯¯;(1)dywherex=g¡1(y)istheinversefunctionofY.Thiscanbeintu-itivelyunderstoodbyobservingthatfX(x)Mx¼Pr(X2(x;x+Mx))¼Pr(Y2(y;y+My))¼fY(y)MyforsmallMxandMy,whereMy=g(x+Mx)¡g(x)dependsong;xandMx.Moreformally:Ifdg=dx>0,then¡¢¡1Pr(g(X)·y)=PrX·g(y)and,ifdg=dx<0,then¡¢¡1Pr(g(X)·y)=PrX¸g(y)(plotg(x)vs.xtoquicklyseethis).1Itneedstobeadi®erentiablefunctionwithdomaincontainedintherangeofX. 21Ifdg=dx>0,thendi®erentiatingZ¡1¡¢g(y)¡1FY(y)=Pr(g(X)·y)=PrX·g(y)=fX(x)dx¡1withrespecttoygives2¡¢dg¡1(y)dx¡1fY(y)=fXg(y)=fX(x);dydyrecallingthatx=g¡1(y).Similarly,ifdg=dx<0,thendi®eren-tiatingZ¡¢1¡1FY(y)=Pr(g(X)·y)=PrX¸g(y)=fX(x)dxg¡1(y)gives¡¢dg¡1(y)dx¡1fY(y)=¡fXg(y)=¡fX(x):dydyTransformation(1)isperhapsbestrememberedbythestatementfX(x)dx¼fY(y)dyandthatfY(y)>0.Thatf(y)=¾¡1f((y¡¹)=¾)forY=¾X+¹and¾>0YXfollowsdirectlyfrom(1)withx=(y¡¹)=¾:¯¯µ¶¯dx¯1y¡¹fY(y)=fX(x)¯¯¯¯=fX:dy¾¾2Rh(y)RecallfromCalculusthat,ifI=f(x)dx,then`(y)@Idhd`=f(h(y))¡f(`(y)):@ydydyThisisaspecialcaseofLeibnitz'rule. 22MoreDistributionsCauchy,afterAugustinLouisCauchy(1789{1857),denotedC(¹;¾),¹2Rand¾2R>0.Thedensityis11fC(x;0;1)=¢¼1+x211andFC(x;0;1)=+arctan(x).2¼Thelocation-scaletransformedversion(moreimportant!)is:11fC(y;¹;¾)=¡y¡¹¢2:¾¼1+¾Usingthesubstitutionu=1+x2,astraightforwardcalculation(youdoit!)revealsthatZZcx1¡¢0x1¡¢22dx=ln1+canddx=¡ln1+c:01+x22¡c1+x22ThisimpliesthatZcxlimdx=lim0=0;c!1¡c1+x2c!1whichwouldseemtoimplythatZZc1xx0=limdx=dx=E[X]:c!1¡c1+x2¡11+x2Thesecondequalityis,however,notingeneraltrue,becausetheorderinwhichtheintegrationisperformedshouldbeirrelevantiftheintegralexists. 23Inthiscase,theorderisconvenientlychosensothatpositiveandnegativetermspreciselycancel,resultinginzero.Moreconcretely,asimilarcalculationshowsthatZkcxlimdx=lnk;c!1¡c1+x2forc>0andk>0.ThisexpressioncouldalsobeusedforR12evaluatingx=(1+x)dx,butresultsinadi®erentvaluefor¡1eachk.Clearlythen,thelatterintegralcannotexist.Asanaside,noticethatf(x)=(1+x2)¡1isanevenfunction,i.e.,itsatis¯esf(¡x)=f(x)forallx(orissymmetricaboutzero).Inthiscase,fiscontinuousforallx,sothat,forany¯nitec>0,ZZc0f(x)dx=f(x)dx:0¡cOntheotherhand,thefunctiong(x)=xisodd,i.e.,satis-¯esg(¡x)=x,and,asgiscontinuous,forany¯nitec>0,RRc0g(x)dx=¡g(x)dx.Finally,ash(x)=f(x)g(x)isalso0¡cRcodd,itfollowsthath(x)dx=0.¡cThus,theaboveresultZcxlimdx=lim0=0;c!1¡c1+x2c!1couldhavebeenimmediatelydetermined. 24Pareto,afterVilfredoPareto(1848{1923),P(®;x0),®;x02R>0;wehave®¡(®+1)fP(x;®;x0)=®x0xI[x0;1)(x)and³´x0®FP(x;®;x0)=1¡I[x0;1)(x):xBesidesitsuseformodelingincomeandotherphenomena,theParetoisanimportantdistributioninthecontextofextremevaluetheory.ThemomentsofXaregivenbyZ1m®¡®¡1+mE[X]=®x0xdxx0®x®¯=0xm¡®¯1m¡®x0®m=x0;m<®;®¡manddonotexistform¸®.Invariousapplications,thesurvivorfunctionofr.v.X,de¯nedbyF¹X(x)=1¡FX(x)=Pr(X>x);(2)isofparticularinterest.ForX»P(®;x),F¹(x)=Cx¡®,where0C=x®.0 25PowerTailsItturnsoutthatthesurvivorfunctionforanumberofimportantdistributionsisasymptoticallyoftheformCx¡®asxincreases,whereCdenotessomeconstant.Ifthisisthecase,wesaythattherighttailofthedensityisPareto{likeorthatthedistributionhaspowertailsorfattails.Somewhatinformally,ifF¹(x)¼Cx¡®,then,asXdF(x)d(1¡Cx¡a)X=/x¡(a+1);dxdxitfollowsfromthemomentsoftheParetodistributionthatthemaximallyexistingmomentofXisboundedaboveby®.ExampleLetthepdfofr.v.Rbegivenbyara¡1fR(r)=a+1I(0;1)(r):(3)(r+1)Toseethatthisisavaliddensity,letu=r=(r+1)(and,thus,¡1¡2r=u=(1¡u),r+1=(1¡u)anddr=(1¡u)du)sothatZZ11a¡1¡(a+1)fR(r)dr=ar(r+1)dr00Z1µ¶a¡1µ¶¡(a+1)u1¡2=a(1¡u)du01¡u1¡uZ1a¯¯1u=aua¡1du=a¯=1:a¯00 26Figure4plotsthedensityfortwodi®erentvaluesof®.1.41.210.80.60.40.20024680.060.050.040.030.020.010020406080100120Figure4:Density(3)for®=0:5(top)and®=10(bottom)Moregenerally,thesamesubstitutionyieldsZ1E[Rm]=ara+m¡1(r+1)¡(a+1)dr0Z1=aua+m¡1(1¡u)¡mdu0¡(a+m)¡(1¡m)=aB(a+m;1¡m)=a;¡(a+1)whichexistsfor1¡m>0orm<1,i.e.,themeanandhighermomentsdonotexist.(Seebelowaboutthegammafunction). 27Forthecdf,notethatµ¶adta¡1¡(a+1)=at(t+1)=fR(t);dtt+1i.e.,µ¶atFR(t)=I(0;1)(t)ort+1µ¶¡at+1=I(0;1)(t):tUsefulFact:Fors¼0,ln(1+s)¼s.ThisisbecauseoftheTaylorseriesexpansion:111¡¢2345ln(1+s)=s¡s+s¡s+Os234Forrelativelylarget,i.e.,farintotherighttail,FR(t)=exp(lnFR(t))¡¡¢¢¡1=exp¡aln1+t¼exp(¡a=t)¼1¡a=t;i.e.,F¹(t)=Pr(R>t)¼a=t/t¡1.RFromthepreviousdiscussiononPareto{liketails,theformofF¹Rsuggeststhatthatmomentsoforderlessthanoneexist,whichagreeswiththedirectcalculationofE[Rm]above.¥ 28ExponentialTailsFormanydistributions,allpositivemomentsexist,suchasforthegammaandnormal.Thesearesaidtohaveexponentialtails.Forinstance,ifX»Exp(¸),thenF¹(x)=e¡¸x.XFigure5showsthesurvivorfunctionforastandardnormalr.v.(left)andtheParetosurvivorfunctionwith®=1(andx0=1).−5x103.50.10.0930.082.50.0720.061.50.050.0410.030.50.0200.0144.555.566.57102030405060Figure5:Comparinganexponentialtail(left)andapowertail(right)Theshapeofeachgraphisthesameforanyintervalfarenoughintothetail.Whilethenormalcdf(orthatofanydistributionwithexponentialrighttail)dieso®rapidly,thecdfwithpowertailstaperso®slowly.Forthelatter,thereisenoughmassinthetailsofthedistributionsothattheprobabilityofextremeeventsneverbecomesnegligible.ThisiswhytheexpectedvalueofXraisedtoasu±cientlylargepowerwillfailtoexist.Thus,inFigure4,whilethepdfclearlyconvergestozeroasxincreases,therateatwhichitconvergesistooslowforthemean(andhighermoments)toexist. 29GammaandBetaFunctionsThebetafunctionisanintegralexpressionoftwoparameters,denotedB(¢;¢)andde¯nedtobeZ1a¡1b¡1B(a;b):=x(1¡x)dx;a;b2R>0:0Closedformexpressionsdonotexistforgeneralaandb;however,theidentity¡(a)¡(b)B(a;b)=¡(a+b)canbeusedforitsevaluationintermsofthegammafunction.Thegammafunctionisanintegralexpressionofoneparameter,denoted¡(¢)andde¯nedtobeZ1a¡1¡x¡(a):=xedx;a2R>0;¡(0):=1:0Thereexistsnoclosedformexpressionfor¡(a)ingeneral,sothatitmustbecomputedusingnumericalmethods.However,¡(a)=(a¡1)¡(a¡1);a2R>1and,inparticular,¡(n)=(n¡1)!forn2N¸2.Forexample,¡(2)=1!=1,¡(3)=2!=2,and¡(4)=3!=6.Thus,thegammafunctionisacontinuousgeneralizationofthefactorialfunctionforN. 30ChiSquarechi{squarewithºdegreesoffreedom,Â2(k)orÂ2,k2R;k>0thedensityisgivenby1k=2¡1¡x=2f(x;k)=2k=2¡(k=2)xeI(0;1)(x)andisaspecialcaseoftheso{calledgammadistribution.Inmoststatisticalapplications,k2N.ThemomentsofX»Â2are,withu=x=2,kZ11sk=2+s¡1¡x=2E[X]=xedx2k=2¡(k=2)0Z11k=2+sk=2+s¡1¡u=2uedu2k=2¡(k=2)01k=2+s=2¡(k=2+s)2k=2¡(k=2)fork=2+s>0,sothat¡(k=2+s)kssE[X]=2;s>¡:¡(k=2)2Inparticular,£¤2E[X]=kandEX=(k+2)ksothatVar(X)=2k:£¤Also,EX¡1=1=(k¡2),k>2. 31Student'stStudent'stwithndegreesoffreedom,abbreviatedt(n)ortn,n2R>0.AfterWilliamSealeyGosset(1876{1937),whowroteunderthenameStudent"whilehewasachemistwithArthurGuinnessSonandCompany.¡1¡¢¡n+1n2f(x;n)=K1+x2=n2;K=¡¢:tnnBn;122Inmoststatisticalapplications,n2N.Ifn=1,thentheStu-dent'stdistributionreducestotheCauchydistributionwhile,asn!1,itapproachesthenormal.Toseethis,recallthatnklim(1+k=n)=e;k2R;n!1or,takingreciprocals,¡k¡ne=lim(1+k=n);n!1which,appliedtothekernelofthedensity(justthepartinvolvingtheargument,andnottheconstantofintegration)gives¡¢¡n+1¡¢¡nlim1+x2=n2=lim1+x2=n2n!1n!1³¡¢¡n´1=22=lim1+x=nn!1hi1=2¡x2¡1x2=e=e2:Thisisenoughtoestablishtheresultbecausealltermsnotinvolv-ingxcanbeignoredbecausethepdfmustintegratetoone. 32MomentsofStudent'stLetT»tn.ThedensityofTissymmetricaboutzero(becausexonlyentersasx2),sothatthemean,ifitexists,mustbezero.Thisistrueforn>1,otherwisethemeandoesnotexist!Ifn>2,thenthevarianceoftnexists,andisgivenbynVar(X;n)=n>2:n¡2Proofs(optional!)1.Fortheexpectedvalue.WewantZ1Z1µ2¶¡ktE[T]=tfT(t;n)dt=Knt1+dt;¡1¡1nwherek=(n+1)=2.Let'signoreKnfornow,andjustlookattheintegral.Splititatzero(becausewehavet2intheintegrand)andletu=t2.Then,pfort<0,thesolutionist=¡u,withdt=¡1u¡1=2du.Also,when2t=¡1(thelowerboundintheintegral)u=1,andwhent=0(theupperbound),u=0.So,Z0µ2¶¡kZ0¡p¢³´¡kµ¶tu1¡1=2t1+dt=¡u1+¡udu¡1n+1n2Z0³´¡k1u=1+du21nZ1³´¡k1u=¡1+du20npSimilarly,fort>0,thesolutionist=+u,withdt=1u¡1=2du,andthe2integralisZ1µ2¶¡kZ1³´¡ktu11=2¡1=2t1+dt=u1+udu0n0n2Z1³´¡k1u=1+du:20n 33Thus,addingthetwopieces,wegetthatE[T]=0,iftheintegralZ1³´¡kuI=1+du0nexists.Now,letv=1+u=n,sothatu=n(v¡1)anddu=ndv,andZ(n1n¯;ifk>1;¡k1¡k¯1k¡1I=nvdv=v=11¡kv=11;ifk·1:Thus,E[T]=0ifk>1,whichisthesameask=(n+1)=2>1,orn>1.2.Forthevariance,westateamoregeneralresultwhichisstraightforwardtoshow:¡¢¡¢£¤k+1n¡k¡¡EjTjk=nk=2¡2¢¡2¢:(F)n1¡¡22Notethat,fromtheargumentofthesecondgammaterminthenumerator£¤kof(F),weseethatEjTjexistsonlyifn>k.hi£¤£¤Fork=2,wehave,forn>2,thatEjTj2=ET2,andET2=Var(T)becauseE[T]=0.Thus,from(F),¡¢¡¢3n¡2¡¡¡2¢¡2¢Var(T)=n1n¡¡¡2¢2¡¢11n¡2¡¡2¡2¢¡¢2¡¢=n1nn¡¡1¡¡1222n1=n2¡12n=:n¡2 34ExpectedShortfallLetRbethereturnona¯nancialassetataspeci¯edtimeinthefuture,suchastheendofatradingday.Theexpectedshortfallisde¯nedtobeE[RjR0th2with(ij)element¾ij,¾i:=¾ii.Importantfactsare1.E[Y]=¹andVar(Y)=§,andthattheparameters¹and§completelydeterminethedistribution.Thus,ifXandYarebothmultivariatenormalwiththesamemeanandvariance,thentheyhavethesamedistribution.2.All2n¡2marginalsarenormallydistributedwithmeanandvariancegivenappropriatelyfrom¹and§;e.g.,¡¢Y»N¹;¾2and,fori6=j,iiiÃ!ÃÃ!Ã!!Y¹¾2¾iiiij»N;:Y¹¾¾2jjijj 523.Animportantspecialcaseisthebivariatenormal,Ã!ÃÃ!Ã!!Y¹¾2½¾¾11112»N;;Y¹½¾¾¾222122whereCorr(Y1;Y2)=½.Thedensityinthebivariatecaseisnottoounsightlywithoutmatrices:½¾X2¡2½XY+Y2fY1;Y2(x;y)=Kexp¡;2(1¡½2)where1x¡¹1y¡¹2K=;X=;Y=:2¼¾¾(1¡½2)1=2¾1¾212¡¢ThemarginaldistributionsareY»N¹;¾2,i=1;2.iii4.IfYiandYjarejointlynormallydistributed,thentheyareindependentifandonlyifCov(Yi;Yj)=0.Forthebivariatenormalabove,Y1andY2areindependenti®½=0.Forthegeneralmultivariatecase,thisextendstonon{overlappingsubsetsY(i)andY(j),i.e.,Y(i)andY(j)areindependenti®¡¢CovY(i);Y(j)=:§ij=0. 535.Fornon{overlappingsubsetsY(i)andY(j)ofY,thecondi-tionaldistributionofY(i)jY(j)isalsonormallydistributed.Thegeneralcaseisgivenlater;inthebivariatenormalcase,theconditionalsare¡¡¢¢¡122Y1jY2»N¹1+½¾1¾2(y2¡¹2);¾11¡½;¡¡¢¢¡122Y2jY1»N¹2+½¾2¾1(y1¡¹1);¾21¡½:P0n6.ThelinearcombinationL=aY=i=1aiYiisnormallyPn0distributedwithmeanE[L]=i=1ai¹i=a¹andvariance(usingz)XnXX002Var(aY)=a§a=aiVar(Yi)+aiajCov(Yi;Yj);i=1i6=jMoregenerally,withAann£nmatrix,itcanbeshownthat,forthesetoflinearcombinations0L=(L1;:::;Lm)=AYwehave0AY»N(A¹;A§A):iidExample:LetXi»N(0;1),i=1;2,andconsiderthejointdensityoftheirsumanddi®erence,Y=(S;D),whereS=X1+X2andD=X1¡X2.LetA=(11)andX=(X;X)»N(0;I),sothatY=AX.1¡11220Fromproperty6above,itfollowsthatY»N(A0;AIA)orY»N(0;2I2),i.e.,S»N(0;2),D»N(0;2)andSandDareindependent.¥ 54ExampleLetXandYbebivariatenormalwith¹1=¹2=0,¾1=¾2=1andcorrelationcoe±cient½=0:5.WewishtoevaluatePr(X>Y+1).Observethat,fromfact6above(i.e.,linearcombinationsofnormalarenormal),X¡Yisnormallydistributed.Next,E[X¡Y]=E[X]¡E[Y],andVar(X¡Y)=Var(X)+Var(Y)¡2Cov(X;Y);sothatX¡Y»N(0¡0;1+1¡2(0:5))=N(0;1):ThusPr(X>Y+1)=1¡©(1)¼0:16:ToevaluateCorr(X¡Y+1;X+Y¡2),notethatCov(X¡Y+1;X+Y¡2)=E[(X¡Y)(X+Y)]£¤£¤22=EX¡EY=0andthusthecorrelationiszero.Fromfacts4and6,weseethatX+YandX¡Yareactuallyindependent.¥ 55SimulationGeneratingasampleofobservationsfromr.v.Z»N(0;In)isobviouslyquiteeasy:EachofthencomponentsofvectorZisiidstandardunivariatenormal,forwhichsimulationmethodsarewellknown.InMatlab,forexample,usetherandncommand.AsY=¹+§1=2ZfollowsaN(¹;§)distribution,realizationsofYcanbeobtainedfromthecomputedsampleszas¹+§1=2z,where§1=2canbecomputedviatheCholeskydecomposition.Forexample,tosimulateapairofmean{zerobivariatenormalr.v.swithcovariancematrix"#1½§=;½1i.e.,withunit{varianceandcorrelation½,¯rstgeneratetwoiidstandardnormalr.v.s,sayz=(z;z)0andthensety=§1=2z.12ThefollowingcodeaccomplishesthisinMatlabfor½=0:6:rho=0.6;[V,D]=eig([1rho;rho1]);C=V*sqrt(D)*V';z=randn(2,1);y=C*z;TogenerateTrealizationsofy,thelastlineabovecouldbere-peatedTtimes,ordoneallatonce"asT=100;z=randn(2,T);x=C*z;whichiseasier,moreelegantandfarfasterinMatlab. 56CDFComputationWiththeabilitytostraightforwardlysimulater.v.s,thecdfPr(Y·y)canbecalculatedbygeneratingalargesampleofobservationsandcomputingthefractionwhichsatisfyY·y.Moregenerally,theprobabilityofanyregioncanbecomputed.Forexample,inthezero{mean,unit{variancebivariatecasewithcorrelation½,theprobabilitythatbothY1andY2arepositive(referredtoasanorthantprobability)canbecomputedwiththefollowingcodeoveragridof½values:rhovec=-0.99:0.01:0.99;n=1000000;emp1=zeros(1,length(rhovec));forloop=1:length(rhovec)rho=rhovec(loop);[V,D]=eig([1rho;rho1]);C=V*sqrt(D)*V';z=randn(2,n);y=C*z;emp1(loop)=length(find(y(1,:)>0&y(2,:)>0))/n;length(y)endplot(rhovec,emp1)TheresultingplotisshowninFigure13. 570.50.450.40.350.30.250.20.150.10.050−1−0.500.51Figure13:OrthantprobabilityPr(Y1>0;Y2>0)forYbivariatenormalversuscorrela-tion½Inthiscase,aclosed{formsolutionexistsfortheorthantprobabil-ity,namely1=4+arcsin(½)=2¼.Morecomplicatedregionsarealsostraightforward:Tocalculate,say,theregiongivenbyY2>0andY10&y(1,:)>y(2,:)))/n;Computingthisrevealsthattheprobabilityispreciselyonehalfoftheorthantprobabilityjustcomputed,whichfollowsbecausethisdensityissymmetricovertheliney1=y2. 58ConditionalDistribution0SupposethatY=(Y1;:::;Yn)»N(¹;§)ispartitionedinto¡¢0twosubvectorsY=Y0;Y0,where(1)(2)00Y(1)=(Y1;:::;Yp)andY(2)=(Yp+1;:::;Yn)with¹and§partitionedaccordinglysuchthat£¤¡¢EY(i)=¹(i);VarY(i)=§ii;¡¢¡¢0i=1;2,andCovY;Y=§,i.e.,¹=¹0;¹0and(1)(2)12(1)(2)23..§11.§12670§=4::::::::::5;§21=§12:..§21.§22Oneofthemostusefulpropertiesofthemultivariatenormalisthatzerocorrelationimpliesindependence,i.e.,Y(1)andY(2)areindependenti®§12=0.Recallthatthecovarianceoftwoindependentr.v.sisalwayszero,buttheoppositeneednotbetrue.Inthenormalcase,however,bothdirectionshold.TheconditionaldistributionofY(1)givenY(2)is¡¢¡¡¢¢¡1¡1Y(1)jY(2)=y(2)»N¹(1)+§12§22y(2)¡¹(2);§11¡§12§22§21: 59Example:Let23023123Y1221167B67C67Y=4Y25»N@415;§A;§=41305:Y30101Becausedet(§)=26=0,Yhasathree-dimensionalpdfandisnotdegenerate.1.ThesixmarginaldistributionsaregivenbyY1»N(2;2);Y2»N(1;3);Y3»N(0;1);"#Ã"#"#!"#Ã"#"#!Y1221Y1221»N;;»N;Y2113Y3011and"#Ã"#"#!Y2130»N;:Y30012.ToderivethedistributionofY2j(Y1;Y3),¯rstrewritethedensityas23YÃ"#"#!267¹(1)§11§124Y15»N;;¹(2)§21§22Y3where23123"#"#31067¹(1)6¢¢¢7§11§1267=67;=41215:¹(2)425§21§220110 60Then,¡¡¢¢¡1¡1Y2j(Y1;Y3)»N¹(1)+§12§22y(2)¡¹(2);§11¡§12§22§21;i.e.,substitutingandsimplifying,£¤¡¢¡1EY2j(Y1;Y3)=¹(1)+§12§22y(2)¡¹(2)"#¡1Ã"#"#!hi21y12=1+10¡11y30=y1¡y3¡1and¡¢¡1VarY2j(Y1;Y3)=§11¡§12§22§21"#¡1"#hi211=3¡10=2;110sothatY2j(Y1;Y3)»N(y1¡y3¡1;2):P33.Thedistributionof(X1;X2),whereX1=i=1YiandX2=Y1¡Y3,isdeterminedbywriting23"#"#Y1X111167X==KY:=4Y25;X210¡1Y30whereKissode¯ned,sothatX»N(K¹;K§K)or"#Ã"#"#!X13102»N;:X2221¥ 61PartialCorrelationDenotingthe(ij)thelementofC=§¡§§¡1§as¾,11122221ijj(p+1;:::;n)thepartialcorrelationofYiandYj,givenY(2)isde¯nedas¾ijj(p+1;:::;n)½ijj(p+1;:::;n)=q:¾iij(p+1;:::;n)¾ijj(p+1;:::;n)Example(cont):Tocompute½13j2,¯rstwrite23YÃ"#"#!167¹(1)§11§124Y35»N;;¹(2)§21§22Y2where232323"#Y1¹12676767¹(1)6Y376¹37607:=E67=67=67¹(2)4¢¢¢54¢¢¢54¢¢¢5Y2¹21and02312323"#Y1¾¾¾211B67C6111312767§11§12:=VarB6Y37C=6¾¾¾7=61107;B67C4313332545§21§22@4¢¢¢5A¾21¾23¾22103Y2sothat"#"#"#211hi¡1hi5=31¡1C=§11¡§12§22§21=¡310=11011and1p½13j(2)=p=3=5:5=3¢1 62Ingeneralterms,¡1C=§"11¡§12§#22§"21#¾11¾13¾12hi¡1hi=¡¾22¾12¾32¾31¾33¾32"#¾¡¾2=¾¾¡¾¾=¾11122213123222=¾¡¾¾=¾¾¡¾2=¾31321222333222and¾13¡¾12¾32=¾22½13j(2)=p(¾¡¾2=¾)(¾¡¾2=¾)111222333222¾22¾13¡¾12¾32=pp¾¾¡¾2¾¾¡¾2221112223332¾22¾13¡¾12¾32=sµ¶sµ¶¾2¾21232¾22¾111¡¾22¾331¡¾22¾11¾22¾33¾22¾13¡¾12¾32=pp¾¾¾¾(1¡½2)(1¡½2)221122331223¾13¾12¾32p¡pp¾11¾33¾22¾11¾22¾33=p(1¡½2)(1¡½2)1223½13¡½12½23=p:(1¡½2)(1¡½2)1223ppUsingthepreviousnumbers,½13=1=2,½12=1=6and½23=0,sothat(4)givespr½13¡½12½231=23½13j(2)=p22=r³¡p¢´=(1¡½)(1¡½)2512231¡1=6asbefore.¥ 63ExampleNeededForTimeSeriesAnalysis0LetY=(Y1;:::Y4)»N(0;§)with231aa2a362716a1aa7§=671¡a24a2a1a5a3a2a1foravalueofasuchthatjaj<1,sothat2323Y1a2a3a1676276Y3716a1aa767»N(0;•);•=67:4Y451¡a24a3a1a25Yaaa212Then,withtheappropriatepartitionsfor¹and•,¡¢0Y1;Y3;Y4jY2»N(º;C):Forthemeanterm,¡¢¡1º=¹(1)+•12•22y(2)¡¹(2)23230a6767¡1¡¢=405+4a5[1]y2¡00a223ay267=4ay25:a2y2 64Forthevariance,¡1C=•11¡•212•22•213231a2a3ahi=1627167¡124a1a5¡4a5[1]aaa1¡a21¡a2a3a1a20232311a2a3a2a2a3=1B62762237C@4a1a5¡4aaa5A1¡a2a3a1a3a3a402311¡a2001B6237C=@401¡aa¡a5A1¡a20a¡a31¡a42310067=401a5:0aa2+1Itfollowsthat¾13j(2)0½13j(2)=q==0;1¾11j(2)¾33j(2)¾14j(2)0½14j(2)=q=p=0;¾¾1+a211j(2)44j(2)and¾34j(2)a½34j(2)=q=p:¾¾1+a233j(2)44j(2)¥ 65TheLinearModel:NotationThelinearregressionmodelrelatesthescalarrandomvariableYtokother(possiblyrandom)variables,x1;:::;xkinalinearfashion,Y=¯1x1+¯2x2+¢¢¢+¯kxk+²;¡¢where²»N0;¾2.AmoreusefulnotationwhichalsoemphasizesthatthemeansoftheYiarenotconstantisYi=¯1x1i+¯2x2i+¢¢¢+¯kxki+²i;i=1;2;:::;n;wherenowadoublesubscriptontheregressorsisnecessary.¡¢Notethat²»N0;¾2,i.e.,theerrortermsareiid.iIfwetakek=1andx1´1,thenitreducestoY=¯1+²,which¡¢isjusttheiidmodelwithY»N¯;¾2.i1Infact,itisusuallythecasethatx1´1foranyk¸1,inwhichcasethemodelissaidtoincludeaconstantorhaveaninterceptterm.Yisalsoreferredtoasthedependent(random)variable,ortheendogenousvariable,whilethekregressorscanalsobereferredtoastheexplanatory,exogenousorindependentvariables.Ifyoucallthemindependentvariables,thenkeepinmindthattheyarenotnecessarily(andrarelyinfact)statisticallyindependentfromoneanother. 66ExampleAssumeademographerisinterestedintheincomeofpeoplelivingandemployedinHamburg.Arandomsampleofnindividualscouldbeobtainedusingpub-licrecordsoraphonebook,andtheirincomesYi,i=1;:::;n,elicited.Assumingthatincomeisapproximatelynormallydistributed,an¡¢unconditionalmodelforincomecouldbepostulatedasN¹;¾2,uuwherethesubscriptudenotestheunconditionalmodelandtheusualestimatorsforthemeanandvarianceofanormalsamplecouldbeused.Amuchmoreprecisedescriptionofincomeisobtainedbytak-ingcertainfactorsintoconsiderationwhicharehighlyrelatedtoincome,suchasage,levelofeducation,numberofyearsofexpe-rience,whetherthepersonismaleorfemale,whetherheorsheworkspartorfulltime,etc.Letxdenotetheageoftheithperson.Aconditionalmodel2iwithaconstantandageasaregressorisgivenbyYi=¯1+¯2x2+²i;¡¢where²»N0;¾2.iTheinterceptismeasuredby¯1andtheslopeofincomeismea-suredby¯2. 67Becauseageisexpectedtoexplainaconsiderablepartofvariabilityinincome,weexpect¾2tobesigni¯cantlylessthan¾2.uAusefulwayofvisualizingthemodelwithk=2iswithascat-terplotofxiandyi.Figure14showssuchagraphbasedona¯ctitioussetofdatafor195individualsbetweentheagesof16and60andtheirmonthlynetincomeinEuro.450040003500300025002000150010005000202530354045505560Figure14:Scatterplotofageversusincome.Itisquiteclearfromthescatterplotthatageandincomearepos-itivelycorrelated.Ifageisneglected,thentheiidnormalmodelforincomeresultsin^¹u=1;797Euroand^¾u=1;320Euro.Fortheconditionalmodel,usingtechniquesdiscussedbelow,theregressionmodelgivesestimatesof¯^1=¡1;465,¯^2=85:4and^¾=755,thelatterbeingabout43%smallerthan^¾u.Themodelimpliesthat,conditionalontheagex,theincome¡¢YismodeledasN¡1;465+85:4x;7552.The¯ttedmodely=¯^1+¯^2xisoverlaidinthe¯gureasasolidline. 68SomeCommentsOurmodelisvalidonlyfor16·x·60.Why?Becauseofthenegativeintercept,smallvaluesofagewoulderro-neouslyimplyanegativeincome.Noticeinthe¯gurethatthelinearapproximationunderestimatesincomebothforlowandhighagegroups,i.e.,incomedoesnotseemperfectlylinearinage,butrathersomewhatquadratic.Howcanweaccommodatethis?Wecanaddanotherregressor,x=x2,tothemodel,i.e.,32Yi=¯1+¯2x2+¯3x3+²i;¡¢where²»N0;¾2and¾2denotestheconditionalvariancebasediqqonthequadraticmodel.Itisimportanttorealizethatthemodelisstilllinear(intheconstant,ageandagesquared).The¯ttedmodelturnsouttobeYi=190¡12:5x2+1:29x3;¾^q=733comparedtooursimpleconditionalmodelYi=¡1465+85:4x2;¾^=755:Thestd^¾qisabout3%smallerthan^¾.The¯ttedcurveisalsoinscribedinFigure14asadashedline. 69LastCommentLookcloseatthe¯gure.Thevarianceofincomeappearstoincreasewithage.Thisisatypical¯ndingwithincomedataandagreeswitheconomictheory.Itimpliesthatboththemeanandthevarianceofincomearefunc-tionsofage.Ingeneral,whenthevarianceoftheregressionerrortermisnotconstant,isissaidtobeheteroskedastic,asopposedtoho-moskedastic.Wewon'tdiscusstestsforthis,orhowtomodelheteroskedasticity(you'vehadthisinyourearliercourses?)butjustmentionsomebookswheregooddiscussionscanbefound:Intriligator,BodkinandHsiao(1995,Ch.7),JohnstonandDinardo(1996,Ch.6),PindyckandRubin¯eld(1997,Ch.6)Greene(2002,Ch.11). 70TimeSeriesRegressionNotationIncertainapplications,theorderingofthedependentvariableandtheregressorsisimportant,becausetheyareobservedintime,andarecalled(bigsurprise)timeseries.Wewillonlyconsiderequallyspacedtimeseries,whichisadequateformanyapplications,butcertainlynotall.Forexample,¯nancialtransactionsonaparticularstockdonotoccuratequallyspacedintervals(thoughwecanmeasurethepriceevery,say,5minutes).Becauseofthis,thenotationYtwillbeused,t=1;:::;T.Thus,thelinearregressionmodelbecomesyt=¯1x1t+¯2x2t+¢¢¢¯kxkt+²t;t=1;2;:::;T;wherexindicatesthetthobservationoftheithexplanatoryvari-itable,i=2;:::k,and²isthettherrorterm.t 71MatrixRegressionNotationThelinearregressionmodelyt=¯1x1t+¯2x2t+¢¢¢¯kxkt+²t;t=1;2;:::;T;canbewrittenmuchmorecompactlywithmatrixnotation!Letxt=(x1t;:::;xkt).Then,instandardmatrixnotation,themodelcanbecompactlyexpressedas23203x11x21¢¢¢xk1¡x1¡676..76x12x22¢¢¢xk27Y=X¯+²;X=4.5=6......7;4...5¡x0¡Tx1Tx2TxkT¡¢where²»N0;¾2I,Yand²areT£1,XisT£kand¯isk£1.Asalreadymentioned,the¯rstcolumnofXis1(acolumnofones).Observethat,intermsofthemultivariatenormaldistribution,¡¢2Y»NX¯;¾I: 72OrdinaryLeastSquares(OLS)Themostpopularwayofestimatingthekparametersin¯isthemethodofleastsquares,whichtakes¯b=argminS(¯),whereXT002S(¯)=S(¯;Y;X)=(Y¡X¯)(Y¡X¯)=(Yt¡xt¯)t=1istheresidualsumofsquares.AssumethatXisoffullrankk.Oneproceduretoobtainthesolu-tionusesmatrixcalculus;ityields@S(¯)=@¯=¡2X0(y¡X¯);settingthistozerogivesthesolution¯b=(X0X)¡1X0Y:Bypre{multiplyingbothsidesby(X0X),noticethat¯bistheso-lutiontowhatarereferredtoasthenormalequations,givenby00XX¯b=XY:IfXconsistsonlyofacolumnofones,then¯breducestothemeanoftheYt.Toseethis,takeX=(11¢¢¢1)0,sothatX0X=TandX0Y=PTt=1Yt.Thus,XT¯b=(X0X)¡1X0Y=T¡1Y=Y:¹tt=1 73AUseful(Projection)MatrixThe¯ttedYtvaluesarede¯nedtobeY^=x0¯bttsothatthevectorof¯ttedvaluesisYb=X¯b0¡10=X(XX)XY=PY;Xwherewede¯ne0¡10P=X(XX)X:XTwoVERYimportantpropertiesofParethatitissymmetric,Xi.e.,P=P0,andidempotent,i.e.,PP=P.Thus,forXXXXXexample,P0P=P.XXXAnymatrixwhichisbothsymmetricandidempotentisreferredtoasaprojectionmatrix.Similarly,theestimatedresidualsaregivenby¡¢²^=Y¡X¯b=IT¡PY=MYXwhere0¡10M:=I¡P=IT¡X(XX)X:XMatrixMisalsosymmetricandidempotent,asyoushouldcheck.AnotherimportantpropertyofMisthatMY=M²;becauseMX=0,whichyoushouldalsoverify. 74MLEof¯Thederivationoftheleastsquaresestimator¯bdidnotinvolveanyexplicitdistributionalassumptions(exceptthatsecondmomentsofthe²iexist.)Ifthemodelis¡¢2Y=X¯+²;²»N0;¾I;thenwemayestimatethek+1unknownparameters¾2and¯,ii=1;:::k,bymaximumlikelihood.Thelikelihoodis½¾2¡T=210fY(y)=(2¼¾)exp¡(y¡X¯)(y¡X¯)2¾2andloglikelihood¡2¢TT¡2¢10`¯;¾;y=¡log(2¼)¡log¾¡(y¡X¯)(y¡X¯)222¾2TT¡¢12=¡log(2¼)¡log¾¡S(¯):222¾2Takingderivativesyields2T1`_=¡X0(y¡X¯)and`_+S(¯):¯2¾2=¡242¾2¾2¾Setting`_¯=0yields00XX¯b=XY;whichisjustthenormalequations.Thus,wegetthesameestima-torfor¯aswithleastsquares! 75Whataboutanestimatorfor¾?AbitmoreworkwiththejointMLEforthek+1parametersrevealsthatS(¯b)2¾~=:TItcanbeshownthattheMLEof¾2isbiased,whileS(¯b)2¾^=T¡kisunbiased.FromtheinvariancepropertyoftheMLE,weknowimmediatelywhattheMLEof¾hastobe.Recallthesimplemodel"inwhichXconsistsonlyofacolumnofones,sothatk=1and¯b=Y¹.Then^¾2reducesto¡¢0¡¢PT¡¢22S(¯b)Y¡X¯bY¡X¯bt=1Yt¡Y¹¾^===;T¡1T¡1T¡1whichweknowfroma¯rstcoursetobeunbiased. 76Whatisthedistributionof¯b?NotethattheOLSestimatorjustamatrix!z}|{¯b=(X0X)¡1X0YisalinearcombinationofY.¡¢RecallthatY»NX¯;¾2I:Thus,¯bisalinearcombinationofanormalrandomvector,sothat,fromourtheoryonthemultivariatenormal,¯bj¾2isalsomultivariatenormallydistributed.Recallthatthe(multivariate)normaldistributionischaracterizedbyitsmeanandvariance(orvariance{covariancematrix).Byexpressingtheestimatorof¯as¯b=(X0X)¡1X0(X¯+²)=¯+(X0X)¡1X0²;(F)weseethat0¡10E[¯b]=¯+(XX)XE[²]=¯:|{z}ZERO!Thatisthemeanofthedistribution,andalsoshowsthat¯bisunbiased. 77Againfrom¯b=(X0X)¡1X0(X¯+²)=¯+(X0X)¡1X0²;(F)notethat¯b¡E[¯b]=¯b¡¯=(X0X)¡1X0²;whichweuseforthevariance.AsVar(²)=E[²²0]=¾2I,£¤£¤Var(¯bj¾2)=E[(¯b¡E¯b)(¯b¡E¯b)0]0=E[(¯b¡¯)(¯b¡¯)]hi=E(X0X)¡1X0²²0X(X0X)¡1=(X0X)¡1X0E[²²0]X(X0X)¡120¡1=¾(XX);sothat³¡¢¡1´¯bj¾2»N¯;¾2X0X: 78GeneralizedLeastSquaresNowconsiderthemoregeneralassumptionthat2²»N(0;¾§);where§isaknown,positivede¯nitevariance-covariancematrix.Thisgeneralizationisextremelyimportant,asitallowsustodealwithheteroskedasticityand/orcorrelation.ThedensityofYisstillmvn,givenby½¾¡T=2¯¯¾2¯¯¡1=210¡1fY(y)=(2¼)§exp¡(y¡X¯)§(y¡X¯):2¾2Let§¡1=2beaCholeskydecompositionof§¡1.Inotherwords,asthechosennotationsuggests,§¡1=§¡1=2§¡1=2:PremultiplyingthemodelY=X¯+²by§¡1=2gives¡1=2¡1=2¡1=2§Y=§X¯+§²;with¡¢¡1=22§²»N0;¾I:Then,usingthepreviousmaximumlikelihoodapproachwith¡1=2¡1=2Y¤:=§Y;X¤:=§XinplaceofYandXimplies¯b=(X0X)¡1X0YJusttheOLSEstimator!§¤¤¤¤¡0¡1¢¡10¡1=X§XX§Y;Justsubstitute!where¯b§isusedtoindicateitsdependenceonknowledgeof§. 79Estimator¯b¡0¡1¢¡10¡1§=X§XX§Yisknownasthegeneralizedleastsquares(GLS)estimate.SimilartotheOLScase,µ³´¶¡1¯b2;§»N¯;¾2X0§¡1X:§j¾ItisattributedtoA.C.Aitken(1934/35),andisthussometimesreferredtoastheAitkenestimator.Anestimateof¾2is2S(¯;y¤;X¤)¾b=T¡kwhere¡¢0¡¢S(¯;y¤;X¤)=y¤¡X¤¯b§y¤¡X¤¯b§¡¢0¡¢¡1=y¡X¯b§§y¡X¯b§:Inpractice,both§isnotknown,butformanymodels,thestruc-tureofitisknown,andisafunctionofasmallnumberofparam-eters. 80ExamplewithanAR(1)ModelArguablythemostbasictimeseriesmodelisanautoregressiveprocessoforderone,orAR(1).Itisgivenbyiid¡¢2²t=a²t¡1+Ut;jaj<1withUt»N0;¾:Wewillshowlaterthatthevariancecovariancematrixofthe²t,t=0;:::;T,isgivenbythe(T+1)£(T+1)matrix231aa2¢¢¢aT676a1a¢¢¢aT¡17167§=6a2a1¢¢¢aT¡27:1¡a266........774.......5aTaT¡1aT¡2¢¢¢1Noticehow§isafunctionofonlyasingleparameter,a.WhydowestartattimeZEROinsteadoftimeONE?Itreallyisarbitrary,buto®erssomenotationalconvenience,andiscommonintheliterature.Thus,wehaveT+1insteadofTdatapoints.Recallthat,fortheGLSestimatorweneeditsINVERSE. 81Itcanbeshownthattheinversetakesonthesimpletridiagonalbandform231¡a0¢¢¢06..76¡ab¡a.767§¡1=60¡a...07;676..74.b¡a500¢¢¢¡a1whereb=1+a2.Evenmoreconveniently,§¡1canbewrittenasC0C,where2p31¡a200¢¢¢0676¡a10¢¢¢0767C=60¡a1......7:6764...0......0750¢¢¢0¡a1 82EstimatingParameteraintheAR(1)ModelImagineforsimplicitythattherearenoregressorsinthemodel,sothatYt=²t.Then½¾¡(T+1)=2¯¯¾2¯¯¡1=210¡1fY(y)=(2¼)§exp¡y§y:2¾2Wehaveanexpressionfor§¡1anditsCholeskydecomposition.ItisstraightforwardtoseethatXT00222yCCy=y0(1¡a)+(yt¡ayt¡1):t=1Whataboutthedeterminant?FromthestructureofmatrixC,¡121j§j=1¡aorj§j=:1¡a2Also,ingeneral,forn£nmatrixAandconstantk,ndet(kA)=kdet(A):Thus,thelikelihoodofYsimpli¯esto1¡(T+1)=221=2L(a;¾;Y)=fY;a;¾(y)=(2¼)(1¡a)¾T+1("#)1XT222£exp¡2y0(1¡a)+(yt¡ayt¡1):2¾t=1Thisissimpli¯ed"becausetherearenomatrices!ExpressionL(a;¾;Y)iscalledtheexactlikelihood. 83YoucanenticeMatlabto¯ndthevaluesofaand¾whichmax-imizesthisexpression,i.e.,^aMLand^¾ML.ThesearetermedtheexactMLE,becauseweusedtheexactlikelihood.Ingeneral,thatisagoodidea,butthereisasimpleapproximationofthelikelihoodwhichisintuitiveandaccurate,whichisquitepopular....TheConditionalMLEoftheAR(1)ModelObviously,inreallife",wedon'tobservetimeserieswithanin¯-nitenumberofobservations,sothattheserieshastohavea¯rstobservation",whichwecalledY0.WhatisthedistributionofY0?ItdependsonthevalueofY¡1(whichdependsonY¡2,etc),allofwhichwedonothave.BecauseobservationsprevioustoY0arenotavailable,wehavetouseitsunconditionaldistribution,whichwewillseelatertobe:µ¶¾2Y0»N0;1¡a2orµ2¶1=2½2¾1¡a1¡a2fY0(y)=exp¡y:2¼¾22¾2 84Thus,theexactlikelihoodcanbewrittenas()¡¢1XT2¡T=22fY0;Y(y0;y)=fY0(y0)¢2¼¾exp¡2(yt¡ayt¡1):2¾t=1TheexactMLEtakesthedistributionofY0intoaccount.ButwhatifwepretendthatY0isNOTarandomquantity,butrathera¯xedvalue?Treatingitassuchmeansthatweareconditioningonitsob-servedvalue.ThenwecanconsiderthelikelihoodofY1;:::;YT,conditionalonY0.Operationally,thismeanswecanignorefY0intheexactlikelihood,givingtheconditionallikelihoodofY=(Y1;:::;YT),()¡¢1XT2¡T=22fY(y)=2¼¾exp¡2(yt¡ayt¡1);2¾t=1whichcanbeusedtogettheconditionalMLEofaand¾.Doestheformoftheconditionallikelihood()¡¢1XT2¡T=22fY(y)=2¼¾exp¡2(yt¡ayt¡1);2¾t=1lookfamiliar?ItispreciselythelikelihoodofaregressionmodelwiththesingleregressorYt¡1andiidnormaldisturbances. 85Thatis,intheoldnotation",theendogenousvectorYandtheexogenousmatrixXaregivenby2323Y1Y067676Y276Y17Y=6..7;X=6..7:4.54.5YTYT¡1Thus,weknowtheOLSestimatorandMLEofaisP0T0¡10XYt=1YtYt¡1baCML=baLS=(XX)XY=X0X=PT¡12t=0YtandPT22t=1(Yt¡baLSYt¡1)¾^LS=:T¡1Whatcouldbemoreintuitiveandeasy?Therelevantquestionis,whatdowelosebyusingtheconditionalMLEinsteadoftheexactMLE?Answer:Notmuchatall.Thetwoareveryclose,anddi®erap-preciablyonlywhenjajisclosetoone.However,asluckwouldhaveit,formanyeconomictimeseries,anAR(1)modelisconsideredtobeareasonableapproximation,andatendstobenearone!BUT!Asfatewouldhaveit,foranearone,eventheexactMLEisconsiderablybiased,andcanbeimproveduponusingmoread-vancedtechniques(whichwewon'tgetinto). 86DistributionoftheMLEAsformostmodels,theexactsamplingdistributionoftheMLEintheAR(1)modelisnotanalyticallytractable,butsimulationo®ersaneasywayofempiricallyapproximatingit.ThiswasdoneforseveralvaluesofaandT,using1,000replica-tions.KerneldensityestimatesofbaMLareshowninFigure15.WithonlyT+1=20observations,thedensityofbaMLisquitespreadout;fora=0,itappearssymmetricaroundzero,whilefora=0:5,thereisanoticeableleftskewness.ThisarisesbecausebaMLisconstrainedtoliebetween¡1and1.Theskewnessisextremeforthea=0:9case,althoughthemodeofthedensityisindeedquitecloseto0.9.Also,itappearsthat,asaincreasesfrom0to1,thevarianceofbaMLdecreases.Thisisveri¯edfromtheplotsforT+1=100andT+1=500.AsTgrows,thedensitybecomeslessskewandverynormal(Gaus-sian)inappearance,centeredonthetruevalueofa.Thevarianceisclearlynotconstantfordi®erentvaluesofa:itdecreasesasamovestowardsone. 8754.543.532.521.510.50−0.6−0.4−0.200.20.40.60.81109876543210−0.6−0.4−0.200.20.40.60.8120181614121086420−0.6−0.4−0.200.20.40.60.81Figure15:KerneldensityestimateoftheMLEofaintheAR(1)modelusing1,000replicationsandvaluesa=0,0.5and0.9.TopisforT+1=20observations,middleisforT+1=100andbottomisforT+1=500. 88AsymptoticDistributionoftheMLEItisoftheoretical(andsometimespractical)interesttoknowhowtheMLEbehavesstatisticallyasT!1.Assumingthatjaj<1,itcanbeshownthatpasy2T(baML¡a)»N(0;1¡a);i.e.,forlargeenoughsamples,baMLisapproximatelynormallydis-tributedwithmeanaandvariance(1¡a2)=T.TheOLSestimatoralsohasthesameasymptoticdistribution.Thisshouldnotbesurprising,astheOLSestimatorcoincideswiththeCMLestimator,whichdi®ersfromtheexactMLEonlybyig-noringthedistributionofthe¯rstobservation.AsTincreases,thein°uenceofthis¯rstobservationgoestozero. 89Toillustratethequalityoftheasymptoticexpressionforthevari-ance,(1¡a2)=T,Figure16showsTtimesthevarianceofbaLSforthreevaluesofT,computedviasimulationbasedon10;000replications,overagridofavalues.Thedottedlineis1¡a2,whichisnearlyreachedforT=100,whileforsmallersamplesizes,thevariancecurveisstillessentiallyquadratic,butlowerinaregionaround0=0andhigheroutside.10.90.80.70.60.50.40.3T=10T=200.2T=1000.10−1−0.500.51Figure16:VarianceofbaLStimesTasafunctionofaItisimportanttokeepinmindthatthisasymptoticresultreliesonthenormalityoftheinnovations.Inpractice,thisassumptioncanbeviolated,whichcouldjeopardizetheusefulnessofthisasymp-toticresult. 90Whathappensasa!1?Becauseoftheaforementionedfactabouteconomictimeserieshavingthepropertythataisnearunity,wenowconsiderwhathappensasaapproachesone.Theexactlikelihoodcannotbeevaluatedata=1becausefY0(¢)=0.Thus,whencomputingtheMLE,theoptimizationalgorithmmustbepreventedfromtryingvaluesofbagreaterthanorequaltoone.ThismotivatesuseoftheleastsquaresestimatorbaLS,i.e.,theconditionalMLE,whichdoesnotrequirefY0.ForthethreesamplesizesT=20,T=100andT=500,andthefourvaluesofa,0:90,0:95,0:99and1:0,themeanandvarianceofbaLSandbaMLbasedon10,000simulatedtimeseriesareshowninTable1,alongwiththepercentageofestimateswhichequalorexceedunity. 91Table1:SmallsamplebehaviorofAR(1)estimatorsnearandonthestationarityborderaaT+10.900.950.991.00OLSMLEOLSMLEOLSMLEOLSMLE200.8310.8330.8870.8890.9500.9500.9190.8321000.8830.8840.9330.9340.9760.9780.9820.970Mean5000.8960.8960.9460.9460.9860.9870.9970.9952020.517.315.812.58.986.6221.621.2ar1002.442.291.561.410.6920.5200.9860.955V¢3105000.4060.3990.2230.2160.06610.06010.04040.0327203.89010.9026.6032.9011000.000.1509.67031.90>%5000.000.000.14032.10aSamplemean(toppanel),1,000timesvariance(middlepanel)andthepercent-ageofestimatesexceeding1.0(bottompanel)ofbaLS(OLS)andbaML(MLE)basedon1,000simulatedtimeseries. 92Inspectionofthetablerevealsseveralfacts:²Notsurprisingly,asthesamplesizeincreases,bothestimatorsimproveintermsofbiasandvariance.²Forallvaluesofa¸0:90,thevarianceofbaMLissmallerthanbaLS.²Forthestationarymodels,thevariancedecreasesasaincreasestowardsone,aswasalsoseeninFigure15.Butwhena=1andTissmall,thevarianceofbothestimatorsjumpsupconsiderably.²Withregardtothebiasoftheestimators,boththeOLSandMLEareextremelydownwardbiasedforT=20andmod-eratelysoforT=100;forT=500,thebiasiszerowhenmeasuredwithtwosigni¯cantdigits.²Forthestationarymodels,thebiasoftheMLEisslightlylessthanthatoftheOLSestimator,withtheirdi®erencebeingmorepronouncedforsmallersamplesizes.²Fora=1,theMLEexhibitsamuchgreaterbiasthantheOLSestimator,particularlyforsmallT.Thisisduetothefactthatbothestimatorshavearelativelyhighvariancewhena=1buttheMLEcannotequalorexceedone.²Avalueofba>1isoflittlepracticalvalueifitcanbeassumedthattheprocessisnotexplosive,inwhichcaseitwillmostlikelybetruncatedtoone.FortherandomwalkwithT=20,ifalloccurrencesofbaLS>1aresettoone,thenthemeanis0.9045(notshowninthetable),sothat,withrespecttobias,thetruncatedOLSestimatorispreferredtotheMLE.Similarresultsholdforthelargersamplesizes. 93TheAR(1)ModelinDetailIfyoucandescribeajobprecisely,orwriterulesfordoingit,it'sunlikelytosurvive.Eitherwe'llprogramacomputertodoit,orwe'llteachaforeignertodoit.(FrankLevy,MITeconomist)3Wehavealreadyde¯nedtheAR(1)modelanddevelopedthetwomethodsofestimation,exactandconditionalMLE,thelatterbeingequivalenttoleastsquares.Indoingso,weusedthevariance{covariancematrixofanAR(1)processwithouthavingderivedit.Wedothatnow.RecallthattimeseriesfYtgissaidtofollowa¯rstorderautore-gressiveprocess,orAR(1)processif,forallt,iid¡¢2Yt=aYt¡1+c+Ut;withUt»N0;¾anda;c2R.TheUtaresometimesreferredtoaswhitenoise,ortheinnovationswhichdrive"theprocess.iid¡¢BecauseweassumethatU»N0;¾2,themodelissometimestreferredtoasaGaussianAR(1),asopposedtoanAR(1)modelwithnonnormalinnovations.TherearetheoreticalreasonswhytheGaussianassumptionshouldbeaccurate,andindeed,manytimeseriesareadequatelymodeledassuch.However,wehaveseenthattherearetimeseriesforwhichitisnotvalid,suchasthedailyreturnson¯nancialassets.3QuotedinDavidWessel'sleadarticleoftheApril2nd,2004WallStreetJournalEuropedealingwiththeoutsourcingofjobsintheUnitedStates. 94DistributionFornow,westickwiththenormalassumption,andconsiderthecovariancesoftheprocess.Let'sapplyrepeatedsubstitutiontothemodelYt=aYt¡1+c+Ut:Then,Yt=aYt¡1+c+Ut=a(aYt¡2+c+Ut¡1)+c+Ut2=aYt¡2+ac+aUt¡1+c+Ut2=a(aYt¡3+c+Ut¡2)+ac+aUt¡1+c+Ut322=aYt¡3+ac+aUt¡2+ac+aUt¡1+c+Ut¡¢322=aYt¡3+c1+a+a+Ut+aUt¡1+aUt¡2..=.Xs¡1Xs¡1sii=aYt¡s+ca+aUt¡ii=0i=0whendonestimes.Withs=t,thisisXt¡1Xt¡1tiiYt=aY0+ca+aUt¡i:(F)i=0i=0IfweconditionalonY0,thenYtisaweightedsumofiidnormalr.v.sandaconstant.Thatmeansthat,conditionalonY0,Ytisalsonormallydis-tributed. 95ConditionalMeanAgain,Xt¡1Xt¡1tiiYt=aY0+ca+aUt¡i:(F)i=0i=0Takethemeanofbothsides,conditioningonY0.Inparticular,notethat"#Xt¡1Xt¡1iiEaUt¡i=aE[Ut¡i]=0;i=0i=0asE[Ut]=0forallt,byassumption.WhataboutthetermXt¡1ia?i=0Ifa=1,thenitisclear.Ifjaj<1,thenitiswell{known,andrealeasytoshow,thatXt¡11¡atia=forjaj<1:1¡ai=0Puttingallthistogether,8Xt¡1j.Thus,recallingourexpressionforthevariance,theconditionalcovarianceis1¡a2(t¡s)as¡a2t¡s2s2Cov(Yt;Yt¡s)=¾a=¾:1¡a21¡a2Inthelimitast!1,weobtaintheunconditionalcovari-ance¾2ajsj°s=limCov(Yt;Yt¡s)=2:t!11¡aFors¸1,thiscanbewrittenas°s=a°s¡1. 101Again,inthelimitast!1,weobtaintheunconditionalcovariance¾2ajsj°s=limCov(Yt;Yt¡s)=2:t!11¡aIncontrasttotheconditionalcovariance,°sdoesnotdependontheparticulartimepointt,onlyonthedistancebetweentwopointsoftime.Thisjusti¯estheuseoftheabsolutevalueofs.Furthermore,ass!1,°s!0.(WHY?)Byusingthe°i,i=0;:::;Ttosetupavariance{covariancematrix,wegetthematrix§,whichweintroducedbefore:231aa2¢¢¢aT676a1a¢¢¢aT¡17167§=6a2a1¢¢¢aT¡27:1¡a266........774.......5aTaT¡1aT¡2¢¢¢1 102MomentsofAR(1):AnotherWayRecallhowwecomputedthemeanofourAR(1)model:(1)Werecursivelysubstitutedttimes,(2)conditioningonY0,wefounditsexpectation,and(3)tookthelimitast!1sothatthee®ectofY0disappeared.Here'sanotherway,assumingtheexistenceofalongrunexpectedvalue.Inthatcase,E[Yt]=E[Yt¡1].Thisisreferredtoasmeanstationary.Let¹=E[Yt].TakingexpectedvaluesofeachterminYt=c+aYt¡1+Utgives¹=c+a¹orc¹=;1¡aasbefore! 103Wecandothesamewiththevariance.Assumingtheprocessisvariancestationary,i.e.,thatVar(Yt)=Var(Yt¡1),wetakethevarianceofbothsidestogetVar(Yt)=Var(c+aYt¡1+Ut):WeknowingeneralthatVar(k+X)=Var(X)forr.v.Xandconstantk.So,Var(Yt)=Var(aYt¡1+Ut):Next,asUtisaniidsequence,itisindependentofYt¡1(andYt¡2,etc.),andweknowthat,ifXandYareindependentr.v.s,Var(X+Y)=Var(X)+Var(Y):Thus,Var(Yt)=Var(aYt¡1)+Var(Ut):Now,recallingthatVar(kX)=k2Var(X),Var(Yt)=Var(aYt¡1)+Var(Ut)22=aVar(Yt¡1)+¾or,with°0=Var(Yt)=Var(Yt¡1),22°0=a°0+¾:Solvingyields¾2°0=:1¡a2 104AndtheCovariances?FirstassumethattheAR(1)modeliscovariancestationary,meaningthatCov(Yt;Yt¡j)=Cov(Ys;Ys¡j)foranytandsandj.Ifyouthinkaboutit,whatthissaysisthatthecovariancebetweenanytwopointsofthetimeseriesdependsonlyonthedistancebetweenthem,notonthecoordinatet.Thatiswhywewrite°j:=Cov(Yt;Yt¡j);and°jis(obviously),afunctiononlyofj,andnotoft.Similartothevariance,thecovarianceisinvarianttoshiftsbyaconstant,i.e.,wecantakec=0withoutlossofgenerality.MultiplyYtbyYt¡j,withj>0,andassumecovariancestationaritytogetYtYt¡j=aYt¡1Yt¡j+UtYt¡jandtakingexpectationsofalltermsgives°j=a°j¡1+E[Ut]E[Yt¡j];whereE[UtYt¡j]=E[Ut]E[Yt¡j]becauseUtandYt¡jareinde-pendent(whenj>0).AsE[Ut]=0,wegetjust°j=a°j¡1j=a°0¾2j=a:1¡a2 105AutocorrelationFunctionThecorrelogramisprobablythemostusefultoolintime-seriesanalysisafterthetimeplot.(Chat¯eld,2001,p.30)RecallthattheunconditionalcovarianceoftheAR(1)processisgivenby¾2ajsj°s=:1¡a2The(unconditional)correlationdividesthecovariancebythestdofthetwocomponents,i.e.,Cov(Yt;Yt¡s)°sjsj½s:=Corr(Yt;Yt¡s)=p==a:Var(Yt)Var(Yt¡s)°0Thesetofvalues½1;½2;:::isreferredtoasthetheoreticalau-tocorrelationfunction,abbreviatedeitherTACForjustACF.Verycommonintimeseriesanalysisistoplotthe½sasaspecialbarplot,andreferredtoasacorrelogram.TwoexamplesofthecorrelogramforanAR(1)processareshowninFigure19.FortheAR(1)model,theshapeoftheACFisquitepredictable,giventheverysimpleformof½s. 1060.50.80.40.60.30.40.20.10.200−0.1−0.2−0.2−0.4−0.3−0.6−0.4−0.8−0.5024681012024681012Figure19:TheoreticalACFoftheAR(1)processwitha=0:5(left)anda=¡0:9(right)HowabouttheSAMPLEACF,i.e.,howcanwecomputethesetofcorrelationscorrespondingtoarealdataset?AssumeanobservedtimeserieshasatotalofTobservations.YouPwouldthinkthatthesamplecounterpartof°is(T¡s)¡1TYY.st=s+1tt¡sItcanbeused,but,fortheoreticalreasons,itismoreadvantageoustouseadivisorofTinsteadofT¡s,i.e.,XT¡1°^s=TYtYt¡s:t=s+1Thisleadstothesampleestimateof½sgivenbyPT°^st=s+1YtYt¡s½^s=Rs==PT:°^0Y2t=1tJustaswiththeTACF,thesampleautocorrelationfunc-tion,orSACF,isgivenbythevectorrandomvariable0R=(R1;:::;Rm):TheupperlimitmcanbeashighasT¡1but,practicallyspeaking,canbetakentobe,say,min(T=2;30). 107TheRicanbeexpressedasratiosofquadraticforms:PTYY0t=s+1tt¡sYAsYRs=P=;(~)TY2Y0Yt=1twherethe(i;j)thelementofAisgivenbyIfji¡jj=sg=2,si;j=1;:::;T.Forexample,withT=5,23231100000000627627610100760001076227627A=6010107andA=6100017;1622726227611761740020254020005110000000022etc.IfthevectorofobservationsYisjointlynormallydistributed(asweoftenassumeinpractice),then(~)isreferredtoasaratioofquadraticformsinnormalrandomvariables.Suchratiosariseinawidevarietyofapplications,andsotheirdistributionalpropertiesareveryimportant.Unfortunately,theirexactpdfandcdfarecomplicatedfunctionswhichcannotrealisticallybecomputed,butapproximationsandnumericalmethodsareavailablewhichcan,andare,implemented. 108Interpretingacorrelogramisoneofthehardesttasksintime-seriesanalysis...(Chat¯eld,2001,p.31)0TheobservedvaluesoftheSACF,r=(r1;:::;rm),computedfora¯niteobservedAR(1)timeseries,willobviouslynotexactlyresemblethecorrespondingTACF,buttheywillbecloseforlargeenoughT.Figure20showstheSACFsoffoursimulatedAR(1)timeseries,eachwitha=0:5andT=50.pThetwohorizontaldashedlinesaregivenby§1:96=Tandpro-videanasymptoticallyvalid95%con¯denceintervalforeachindi-vidualrs.0.60.50.40.40.30.20.20.100−0.1−0.2−0.2−0.3−0.4−0.4−0.5−0.60246810120246810120.50.50.40.40.30.30.20.20.10.100−0.1−0.1−0.2−0.2−0.3−0.3−0.4−0.4−0.5−0.5024681012024681012Figure20:TheSACFsoffoursimulatedAR(1)timeserieswitha=0:5andT=50 109Weseethat,forsamplesizesaroundT=50,theSACFdoesnotstronglyresembleitstheoreticalcounterpart.Figure21issimilarbutusesT=500observationsinstead.TheSACFisnowfarclosertotheTACF,butcanstilltakeonpatternswhichdi®erfromthetruevalues.0.60.50.40.40.30.20.20.100−0.1−0.2−0.2−0.3−0.4−0.4−0.5−0.60246810120246810120.50.60.40.40.30.20.20.100−0.1−0.2−0.2−0.3−0.4−0.4−0.5−0.6024681012024681012Figure21:TheSACFsoffoursimulatedAR(1)timeserieswitha=0:5andT=500 110UnknownMean:RegressionResidualsInpractice,themeanofthetimeseriesYisnonzeroandunknown.Thesamplemeancan(still)beusedtoestimatethemeanoftheseries.Then,thesamplecovarianceisinsteadcomputedasXTXT¡1¡1°^s=T(Yt¡¹^)(Yt¡s¡¹^)=T²^t²^t¡st=s+1t=s+1and0b²Asb²Rs=0;b²b²whereb²=(^²;:::;²^)0=Y¡¹^aretheestimatedresiduals1Tand^¹isthesamplemeanoftheYt.RecallthatthemodelYt=¹+²tisaspecialcaseofthelinearregressionmodel,Y=X¯+²,withestimatedresidualsY^¡X¯^=MY=M²:Thus,thesthelementoftheSACFcanbewrittenasb²0Ab²²0MAM²²0MAM²sssRs=b²0b²=²0M0M²=²0M²;recallingthatMissymmetricandidempotent,explainingwhyM0M=M. 111Justwhatexactlyisthemodelnow?WenowhavearegressionmodelY=X¯+²(})wheretheresiduals²arenotiid,butratherfollowanAR(1)pro-cess.Thatis,²t=a²t¡1+Ut;(Ä)iid¡¢andtheU»N0;¾2.tEquation(})iscalledtheobservationequation(becauseyouobserveYtandknowX),while(Ä)iscalledthelatentequation.Accordingtothedictionary,latentisanadjectivewhichmeanstoliehidden".AkintoGreeklanthanein:toescapenotice.Themodelcanalsobewrittenas231aa2¢¢¢aT676a1a¢¢¢aT¡17¡¢167Y»NX¯;¾2§;§=6a2a1¢¢¢aT¡27:1¡a266........774.......5aTaT¡1aT¡2¢¢¢1Ifaisknown,howdoweestimate¯and¾?If¯isknown,howdoweestimateaand¾? 112EstimationIIfaisknown,then§isknown,andwecanuseGLS.If¯isknown,thenX¯isknown,andwecansetR=Y¡X¯andthenapplyourexactorconditionalMLEtotheseriesRtogetanestimateforaand¾.Butwhatifalltheparametersareunknown?Canwesomehowcombine"thetwopreviousprocedures?Yes.Wecaniteratebetweenthem.1.Firstsetato,say,zero.Callthis^a0,wherethesubscriptindicatesthatwewillhaveasequenceofestimatorsfora.2.Basedon^a0,computetheGLSestimator,called¯b0.As^a0=0,thisisthesameastheOLSestimator.3.Basedon¯b0,computetheresiduals,sayR1=Y¡X¯b0,andusethemtoestimatea(witheitherexactorconditionalMLE),andcallit^a1.4.Basedon^a1,computetheGLSestimator¯b1.5.Basedon¯b1,computetheresidualsR2=Y¡X¯b1,andusethemtocompute^a2.6.Keeprepeatingthesetwostepsuntilconvergence". 113EstimationIIThepreviousmethodiselegantandrelativelysimple.Nevertheless,thereareadvantagestoestimatingalltheparameterssimultaneously.Onemightthinkthatthisisdi±cult,butitisnotatall:Theexactlikelihoodoftheobservationsisnothingbutthedensityofa¡¢X¯;¾2§vectorrandomvariable,whichiseasytoevaluate.N¡¢SotheMLEisjustthevalueofµ=¯0;¾2;awhichmaximizethelikelihood,orargmaxL(µ;Y):µThatmightseemeasiersaidthandone:Wehavetomaximizeafunctionwithrespectto2+kvariables!Buttherearenumericalmethods(whichuseresultsfrommultivariatecalculus)designedtodothis,andtheyareimplementedasblackboxes"inMatlab,Splusandothersuchprograms.Thus,itisstraightforwardtocalculatetheexactMLE.Abene¯tofthismethodofestimationcomparedtotheprevi-ousoneisthat,asaby{productofmaximizingthelikelihood,wealsogetapproximationsforthestandarderrorsoftheparameters,whichcanbeusedtoformcon¯denceintervalsorconducthypoth-esistests. 114Thereis,however,aproblemwiththismethod:ForlargeT,thematrixcalculationsinvolvedinevaluatingthenormaldensitywillmakethemethodslow.Therearemoree±cientmethodswhichcanevaluatethelikelihoodwith(much)lesscalculation.OnesuchmethodusestheKalman¯lter.Wewillnotstudysuchtechniquesinthisclass.Withtheavailablecomputingpower,youwillbarelyevennoticetheslownessof(comparedtofastermethods)forTlessthan,say,200,butasTapproaches,say,1,000,evenfastcomputerswilltakequitesometime.Asamplesizeof1,000mayseemridiculoustoamacroeconometri-cian,whoisusedtoyearlydatasince1920,or,atbest,monthlydataoverthepast30years.Butdaily¯nancialdatagoesbackmanyyears,sothatseveralthousandobservationsarenotuncommon,andwithultrahighfrequencydata,youmight¯ndyourselfwithahundredthou-sandobservations!AnothermethodwhichsavesenormouslyoncomputationaltimewhenTislargeistomaximizeanapproximatelikelihood.Thistypicallyinvolvesconditioningonthe¯rstseveralobservations,justlikewedidwiththepureAR(1)model.WewillseehowtodothislaterinthelargercontextofARMAandARMA{GARCHmodels. 115ForecastingtheAR(1)ModelTheonlyusefulfunctionofastatisticianistomakepre-dictions,andthustoprovideabasisforaction.(WilliamEdwardsDeming)Oneofthemostinterestingandusefulaspectsoftimeseriesanal-ysisisextrapolatingthemodelbeyondthesamplesizeTtoobtainpointandintervalestimatesofvalueswhichwillbeobservedatalaterdate.Thisisreferredtoasforecasting,sometimesdistinguishedfromprediction,althoughwewillusethesetermsinterchangeably.AssumevaluesY1;:::;YTareavailable,whichwerefertomoregenerallyastheinformationsetuptotimeT,denoted•T.AsYT+1=aYT+UT+1,alogical(andoptimal)forecastofYT+1given•TisaYT,obtainedbyreplacingtheunobservablevalueofUT+1byitsexpectedvalue.WedenotethisasYT+1j•Tor,morecommonly,asYT+1jT. 116MeasuringForecastQualityHowgoodisYT+1jT?Themostcommonmeasuresareitsmean,varianceandmeansquarederror,ormse,whichisde¯nedby£¤2mse(µ^):=E(µ^¡µ):Itcanbeshownthat¡¢2mse(µ^)=Var(µ^)+Bias(µ^);wherethebiasofµ^isjustµ^¡µ.AsYT+1jT¡YT+1=aYT¡(aYT+UT+1)=¡UT+1;itfollowsthat£¤EYT+1jT¡YT+1=0and¡¢¡¢2VarYT+1jT¡YT+1=mseYT+1jT=¾:Doyouseeanyproblemswiththis? 117Toobad,thataisnotknown...It'stoughtomakepredictions,especiallyaboutthefuture.(YogiBerra)Asawillalmostalwaysbeunknown,itisreplacedbyanestimate,say^a=^aML,togetY^T+1jT:=^aYTand¡¢h¡¢2ih¡¢2imseY^T+1jT=EY^T+1jT¡YT+1=EY^T+1jT¡aYT+aYT¡YT+1h¡¢ih¡¢iY^22=ET+1jT¡aYT+EaYT¡YT+1+crosstermh¡¢iY^22=ET+1jT¡aYT+¾;wherethecrosstermh¡¢¡¢i2EY^T+1jT¡aYTaYT¡YT+1=2E[((^a¡a)YT)(¡UT+1)]iszerobecauseUT+1isindependentof(^a¡a)YT.Asthecorrelationbetween^aandYTgoestozeroasthesamplesizeincreases,itfollowsthath2i£¤h2i¾21¡a2¾22E((^a¡a)YT)¼EYTE(^a¡a)¼2=;1¡aTTsothat¡¢¡¢mseY^¼1+T¡1¾2;T+1jTwhichisindependentofthetruevalueofaandisasymptotically¡¢identicaltomseYT+1jT.Ofcourse,inpractice,¾2isunknown,andisreplacedbyanesti-mate. 118Figure22showsthemseofY^T+1jTforthismodelasafunctionof(positivevaluesof)a,computedviasimulationusing100,000replications,basedon¾2=4andT=10.5MLEOLSYW4.94.84.74.64.54.44.34.200.20.40.60.81Figure22:Themseofone{stepaheadforecastY^T+1jT=^aYTfortheAR(1)modelasafunctionofautoregressiveparametera,with¾=2.SolidismseforbaML;dashedisbaLSanddash{dotisbaYW.ThiswasdoneforthethreeestimatorsexactML,denotedbaML;leastsquares,denotedbaLS;andbaYW,whichisreferredtoastheYule{Walkerestimator,de¯nedasPTt=2YtYt¡1baYW:=P:TY2t=1tItwillbeformallyintroducedanddiscussedlater.Fornow,justnoticethatbaLSandbaYWarealgebraicallyveryclose.The¯rstobservationtobemadefromFigure22isthat,withrespecttomse,theexactMLEissuperiortotheconditionalMLE(baLS)forall00.Ingeneral,itistruethatanAR(p)modelisnotstationaryifXpai¸1:i=1Fortheunconditionalvariance,°0=limt!1Var(Yt)andtheun-conditionalcovariances,°s=limt!1Cov(Yt;Yt¡s).ForthevarianceandcovariancesintheAR(1)case,weagreedthatwecouldjustsetctozerowithoutlossofgenerality.ThisisstilltrueintheAR(p)case,butlet'sleaveitinanyway.Itisstraightforwardtoshow(anditisHOMEWORK)thatYt¡¹=a1(Yt¡1¡¹)+a2(Yt¡2¡¹)+¢¢¢+ap(Yt¡p¡¹)+Ut;where¹isgivenabove. 122Moments:CovariancesMultiplyingbothsidesbyYt¡¹,takingexpectationsandusingtherelation°s=°¡sgives2°0=a1°1+a2°2+¢¢¢+ap°p+¾:Similarly,multiplyingYt¡¹=a1(Yt¡1¡¹)+a2(Yt¡2¡¹)+¢¢¢+ap(Yt¡p¡¹)+UtbyYt¡j¡¹andtakingexpectationsgives°j=a1°j¡1+a2°j¡2+¢¢¢+ap°j¡p;j=1;2;::::ThesearereferredtoastheYule{WalkerequationsfromYule(1927)andWalker(1931).Dividingby°0yields½j=a1½j¡1+a2½j¡2+¢¢¢+ap½j¡p;j=1;2;:::;whicharealsocalledtheYule{Walkerequations.Using¯rstp+1Yule{Walkerequations(startingwith°0),wegetasystemofp+1equationsinp+1unknowns,whichcanbesolvedfor°0;:::;°p.Once°0;:::;°parecomputed,higherordercovariancesfollowdi-rectlyfromtheYule{Walkerequations. 123ExamplewithAR(2)ModelWithp=2,2°0=a1°1+a2°2+¾°1=a1°0+a2°1°2=a1°1+a2°0;whichcanbesolvedtoyield¡¢a2+a¡a22(1¡a2)2a12122°0=¾;°1=¾;°2=¾;DDDwhere5D=(a2+1)(a1+a2¡1)(a2¡a1¡1):Tocompute°jforj¸3,usetherecursion°j=a1°j¡1+a2°j¡2.Thecorrelations½j=°j=°0are,inthiscase,aa2+a¡a21122½0=1;½1=;½2=1¡a21¡a2and½j=a1½j¡1+a2½j¡2forj¸3.5NotethatD=0ifa+a=1,whichwesaidearliercannotbethecaseifthemodelistobe12stationary. 124CovariancesforplargeWhilesolvingthesetofcovarianceequationstoobtainthe°sisclearlyfeasibleforanyp,inpractice,thecalculationsbecomealge-braicallymessyforp¸4andnumericallyprohibitiveaspgrows.Instead,severalothermethodshavebeendevelopedforcalculatingthe°swhicharefarmoreexpedient.AnalgebraicallyattractivemethodwhichalsolendsitselftonumericcomputationwasgivenbyvanderLeeuw(1994),whoseresultbuildsontheworkofGal-braithandGalbraith(1974),McLeod(1975)anddeGooijer(1978),andissimilartoMittnik(1988).Wecan,withoutanylossofgenerality,take¾2=1becauseitisjustascaleparameter.Nowlet§denotetheT£TunconditionalcovariancematrixofththeYj,i.e.,the(ij)elementof§is°i¡j,forT¸p+1.VanderLeeuw(1994)showedthat§¡1=P0P¡QQ0;wherePistheT£Tlowertriangularbandmatrixwith¯rstcolumn0(®0;®1;:::;®p;0;:::;0),®0=1,®i=¡ai,i=1;:::;p,and23"#®p®p¡1¢¢¢®167T260®p¢¢¢®27Q=;T2=6.........7;04...500¢¢¢®pandQisofsizeT£p.HOMEWORK:Compute§fortheAR(2)modelbasedonvanderLeeuw'sformula. 125StationarityandUnitRootProcessesIntheAR(1)case,thea(L)polynomialisjusta(L)=1¡a1L.WhenListreatedlikeavariable,therootsofa(L),i.e.,thesolu-tiontotheequationa(L)=0,is1=a1.RecallthattheAR(1)modelisstationaryifjaj<1.Thus,thestationarityconditioncanbestatedasrequiringthattherootoftheAR(1)polynomialisgreaterthanone.ThiscarriesovertotheAR(p)case:Themodelisstationarywhenthemodulusofallp(possiblycomplex)rootsofthepolynomialpa(L)=a(L)=1¡a1L¡¢¢¢¡apLaregreaterthanone.Letpa(L)=1¡a1L¡¢¢¢¡apL=(1¡¸1L)¢¢¢¢¢(1¡¸pL)sothattherootsofa(L)aregivenby¸¡1;:::;¸¡1.1p¯¯Themodelisstationarywhen¯¸¡1¯>1or,equivalently,whenij¸ij<1,i=1;:::;p,wherej¸ijisthemodulusof¸i.Ifthecomplexnumbersareplottedintheusualfashion,thisisequivalenttorequiringthattherootslieoutsideofthecomplexunitcircle. 126TherootsareeasilycomputedinMatlabusingthebuilt{inrootsfunction.Forexample,ifthemodelisanAR(2)withparametersa1=1:2anda2=¡0:8,thenexecutingrr=roots([0.8-1.21])returnsthetwocomplexroots0:75§0:8292i.Themodulusiscomputedasabs(rr),givinginthiscase1.1180forbothroots,sothatthemodelisstationary.Example:Forthespecialcasewithp=2,thequadraticformulacanbeused:TreatingLasavariablegives2a(L)=1¡a1L¡a2L=(1¡¸1L)(1¡¸2L):MultiplyingthisbyL¡2andsetting¸=L¡1gives2¸¡a1¸¡a2=(¸¡¸1)(¸¡¸2);withrootsq11¸=a§a2+4a1;211222¡1¡1sothat¸and¸arerootsofa(L).12Fromthis,italsofollowsthattherootsarecomplexifa2+4a<0,12andrealotherwise. 127UnitRootsIfj¸ij<1,i=1;:::;p¡1and¸p=1,thentheAR(p)processa(L)Yt=c+Utissaidtohaveaunitrootandwillresemblearandomwalk,possiblywithdrift.As¸p=1,theprocesscanbewrittenas(1¡¸1L)¢¢¢¢¢(1¡¸p¡1L)(1¡L)Yt=c+Utor,withZt=(1¡L)Yt=Yt¡Yt¡1,as(1¡¸1L)¢¢¢¢¢(1¡¸p¡1L)Zt=c+Ut:Thatis,the¯rstdi®erenceofanAR(p)processwitha(single)unitrootisastationaryAR(p¡1)process.Thepracticalimplicationofthisfactisthat,whenfacedwithtimeseriesdatawhichresemblearandomwalk,the¯rstdi®erenceZt=Yt¡Yt¡1shouldbecomputedandtheZtshouldbesubsequentlyanalyzed.503201−500−100−1−150−2−200−3−250−4050010001500200025003000020406080100Figure23:Simulatedrandomwalkprocess,Yt,(left)and¯rst100observationsofthe¯rstdi®erenceseriesZt=Yt¡Yt¡1(right) 128ExamplewithanAR(4)ModelForanAR(1)process,wehaveaunitrootifa1=1.Ingeneral,ifPpi=1ai´1,thenthereisexactlyoneunitroot.TheAR(4)modelYt=¡0:2Yt¡1+1:1Yt¡2+0:4Yt¡3¡0:3Yt¡4+Utissuchthata1+a2+a3+a4=1.Themodelcanalsobewrittenas(1¡¸1L)(1¡¸2L)(1¡¸3L)(1¡L)Yt=Ut;¡1¡1withj¸j=2:4839,andj¸j=1:1584(to4decimalplaces).12;3Denotethe¯rstdi®erenceofYtasXt=(1¡L)Yt,sothat(1¡¸1L)(1¡¸2L)(1¡¸3L)Xt=UtisastationaryAR(3)process.Multiplyingout,themodelforXtcanbewrittenasXt=¡1:2Xt¡1¡0:1Xt¡2+0:3Xt¡3+Ut:AsimulatedrealizationofYtisshownintheleftpanelofFigure24.Theoverallmovement"indeedhastheusualcharacteristicofarandomwalk,butthereisalsoshort{termpersistencebecauseofthestationaryAR(3)component. 129256420215010−25−40−60100200300400500050100150200Figure24:Simulatedunit{rootAR(4)process(??)(left)andpartofthe¯rstdi®erenceseries(right)Noticetheabruptandpersistentchangeofdirection"ofthese-riesaroundobservation350.Thisispurelyanartifactofchance,thoughwhenfacedwithsimilar{lookingrealdata(say,astockpriceorexchangerate),anaturalinclinationwouldbetoconsiderthechangetobeblatantevidence"ofastructuralbreakinthemodelpurportedtodescribetheevolutionoftheseries.Therightpanelshowsthe¯rst200observationsofXt=(1¡L)Yt.Whilethisseriesappearsmeanstationary(aroundzero),onemightfeelbehoovedtoquestion(andstatisticallytest)theconstancyofthevariance,whichappearstochangewithtime.Thereasonfortheseemingvolatilityclustering"ofXtisthelargenegativecoe±cientonXt¡1andtheproximityofthepolynomialrootstotheunitcircle.Forexample,asimilare®ectisrealizedwiththeAR(1)modelwithcoe±cient¡0:99. 130LeastSquaresEstimatorAsintheAR(1)case,theOLSestimatorisapplicable.AlsoasintheAR(1)case,itcoincideswiththeconditionalMLE,i.e.,maximizingjustfYjY0(yjy0)=f(Y1;:::;YT)j(Y1¡p;Y2¡p;:::;Y0)(yjy0)insteadoffY0(y0)£fYjY0(yjy0).TheOLSestimatorisa(matrix)closedformsolutionnotrequiringstartingvalues,andistriviallycomputed.Thisisobtainedbytakingtheendogenousvariabletobe(Y;:::;Y)0andtheT£p1Tdesignmatrix23Y0Y¡1:::Y1¡p676Y1Y0:::Y2¡p766......77...Z=6......7;676...7674YT¡2YT¡3:::YT¡p¡15YT¡1YT¡2:::YT¡pi.e.,00¡100baLS(p)=(^aLS(1;p);:::;a^LS(p;p))=(ZZ)Z(Y1;:::;YT):Whenitisclearfromthecontext,wewillsuppresstheexplicitdependenceoftheestimatoronpandjustwritebaLS.Writingthisout,baLSis2PPP3¡12P3T¡12T¡2T¡pT6i=0Yii=¡1YiYi+1¢¢¢i=1¡pYiYi¡1+p76i=1YiYi¡1766PT¡1PT¡22PT¡p7766PT776i=0YiYi¡1i=¡1Yi¢¢¢i=1¡pYiYi¡2+p76i=1YiYi¡276......76..7:6...76.74545PPPPT¡1T¡2T¡p2Ti=0YiYi+1¡pi=¡1YiY2+i¡p¢¢¢i=1¡pYii=1YiYi¡p 131Forclarityofstructure,withp=3thisis2PPP3¡12P3T¡12T¡2T¡3T6i=0Yii=¡1YiYi+1i=¡2YiYi+276i=1YiYi¡1766PT¡1PT¡22PT¡37766PT77:4i=0YiYi¡1i=¡1Yii=¡2YiYi+154i=1YiYi¡25PPPPT¡1T¡2T¡32Ti=0YiYi¡2i=¡1YiYi¡1i=¡2Yii=1YiYi¡3Byitsmatrixconstruction,Z0Zissymmetric,whichisalsoseenbylookingattheindividualelementsinthiscase.However,notethatalltheelementsalonganychosendiagonalofZ0Zareclose,butnotidentical.Theestimateof¾2iscomputedasusual,i.e.,thesumofsquaredresidualsdividedbyeitherT(fortheconditionalMLE)orbyT¡ptoadjustforbias.TheOLSestimatorwillyieldestimateswhicharereasonablyclosetotheexactMLEvalues(exceptforverysmallsamplesizesand/orcasesforwhichoneormorerootsoftheprocessareclosetothestationarityborder).Asasymptotically,the¯rstpobservationsbecomenegligible,onewouldexpectthatbaLShasthesameasymptoticdistributionastheMLE.Thisistrue,andwasshownbyMannandWold(1943),pasy¡¢i.e.,T(ba¡a)»N0;¾2¡¡1(seealsoHamilton,1994,pp.LS215-6;andFuller,1996,x8.2.1). 132Yule{WalkerTheYule{Walkerequationscanalsobeusedtocomputereliablestartingvaluesoftheai,denoted^aYW(i;p),i=1;:::;p,forp1.Becauseitismoreconvenienttoworkwithconvergentseries,theMA(1)modelwithjbj<1ispreferredtojbj>1.Ifjbj<1,themodelissaidtobeinvertibleandnoninvertibleotherwise.Ifb=1,thereisonlyonerepresentationofthemodel,butasthesumofcoe±cientsin(8)diverges,theborderlinecaseofb=1isalsodeemednoninvertible. 136Fromthesimplecovariancestructure,matrix§anditsinverseareeasilyconstructed,sothatthelikelihoodand,thus,theMLEcanbecomputed.Unfortunately,anexpressionresemblingtheAR(p)case,whichpartitionsthelikelihoodisnotavailablewiththemovingaveragestructure.Onewayofproceedingisjusttocompute§corre-spondingtothewholeT{lengthsample,alongwithitsinverseanddeterminant,andthenevaluatethelikelihoodfunctionas¯¯§¯¯1=2½¾¡110¡1L=exp¡y§y;(9)(2¼¾2)T=22¾20wherey=(y1;:::;yT).The¯rstproblemassociatedwiththenumericmaximizationofthelogof(9)isthat^bshouldberestrictedtoliein(¡1;1]sothattheresultingmodel(exceptattheborderline)isinvertible.Thiscanbeachievedinthefollowingway.Let¾2beanypositivevalue,01,setbÃ1=bandthen¾2þ2=b2. 137Thesecondproblemwithnumericmaximizationof(9)isthatitwillbecomputationallyslowwhenTisover,say,100,andwillbeessentiallyimpossibletocomputefordatasetswiththousandsofobservations,asarise,forexample,intheanalysisof(daily,orevenhigherfrequency)returnson¯nancialassets.Awayaroundthisproblemistouseanapproximationtothelikelihoodfunctioninplaceoftheexactexpression.IfweconditiononU0´0=E[U0],thenthesimplerecursionUt=Yt¡bUt¡1,t=1;:::;Tcanbeusedtogetthe¯ltered"valuesU^t=Yt¡bU^t¡1.TheconditionallikelihoodisthenjusttheproductofTnormaldensitiesevaluatedattheU^t,i.e.,¡¢YT³´Lcondb;¾2;Y;U=ÁU^;0;¾2;0tt=1½¾¡¢11u22pÁu;0;¾=exp¡;2¼¾2¾2anditslogwouldbemaximizedoverband¾. 13843.50.230.152.520.11.510.050.5000.20.30.40.50.60.70.80.916810121416Figure25:DensityoftheexactMLE(solid)andconditionalMLE(dashed)oftheMA(1)modelbasedonT=15observations,b=0:9and¾=10.Leftplotisfor^bandrightisfor^¾.Itshouldbeclearthat,asymptotically,thetwomethodsoflikeli-hoodcalculationareequivalentwhenthemodelisinvertible.How-ever,therewillbeconsiderabledi®erenceswhenworkingwithsmallsamplesizes.Figure25showstheresultsofasimulationstudy,basedon1,000replications,ofthebehavioroftheexactandcon-ditionalMLEofanMA(1)modelwithparametersb=0:9and¾=10andsamplesizeT=15.WeseethattheexactMLEforbisconsiderablyclosertothetruevalue,buthasatendencytopileup"ontheborderlinevalueof1.0.For¾,theexactMLEhasasmallervariancethantheconditionalvalue,buttheconditionalMLEislessbiased.Asimilarrunbutusingb=0revealsthattheexactandconditionalMLEarevirtuallyidentical.Thisisawell{knownphenomenon:Theclosertheparameteristotheinvertibilityborder,themoreaccuratetheexactMLEiscomparedtotheconditionalMLE. 139Toinvestigatethisinabitmoredetail,thebiasandmseoftheexactandconditionalMLEwerecomparedoveragridofb{valuesforT=15,¾=1andbasedon10,000replications.FromthegraphsinFigure26,wesee,somewhatsurprisingly,thattheconditionalMLEisactuallypreferredintermsofbothbiasandmseformostoftheparameterspace.Forjbj>0:7,exactMLEexhibitslowerbias,whileforjbj>0:8,italsohassmallermse.0.20.140.150.120.10.050.100.08−0.05−0.10.06−0.15−0.20.04−1−0.500.51−1−0.500.51Figure26:Thebias(left)andmse(right)oftheexactMLE(solid)andconditionalMLE(dashed)of^bintheMA(1)modelasafunctionofparameterb,basedonT=15observations,¾=1and10,000replications.Analternativewayofcomputingtheexactlikelihoodistocastthemodelintheso{calledstatespacerepresentationandusethemethodswellknowninengineeringandsystemstheory,mostnotablytheKalman¯lter.ThisapproachavoidstheconstructionandinversionofT£Tmatricesandisthusfarfastertocompute.CombinedwiththegeneralityofthestatespacemodelandthefactthattheKalman¯lterlendsitselfwelltocomputercomputation,thisapproachisbyfarthemostattractiveandhasbecomeastandardtoolinstatisticaltimeseriesanalysisandeconometrics.Afulltreatmentrequiresatleastseveralpages,andweinsteadpointthereadertothenumerousbookswhichdetailitsconstruction.AparticularlyreadablepresentationandemphasisonunivariatetimeseriesanalysiscanbefoundinHamilton(1994),whileBrockwellandDavis(1991)andShumwayandSto®er(2000)botho®erchaptersonstatespacemodelingandKalman¯ltering.ThebooksbyWestandHarrison(1997)andDurbinandKoopman(2001)onstatespacemodelsareaimedatstatisticiansandarehighlyrecommended.Finally,thebookbyChuiandChen(1999)isverycompleteandaccessible,butisaimedmoreatengineers. 140Oneissuenotdiscussedishowtochoosestartingvaluesof^band¾forthelikelihoodmaximization.Forsuchamodel,havingonlytwoparameters,startingvaluesarenotveryimportant:whenusingjustthenaivevalueofzeroforbandthesamplevariancefor¾2,oneortwoiterationsofthenumericfunctionmaximizationalgorithmareenoughtopullthevaluesintoaregionveryclosetothe¯nalvalues.Nevertheless,itis,ingeneral,betterpracticetousemoreintelligentstartingvaluesifeasilycomputedonesareavailable.FortheMA(1)model,(7)canbeusedtoderiveamethodofmomentsestimatorwhichistriviallycomputed:Itssolutionsareb=0if½1=0and1¡p¢2b=1§1¡4½1;½16=0:(10)2½1Thisisvalidbecausej½1j<1=2foranMA(1)model.Toensure¯¯¯^b¯<1,i.e.,aninvertiblemodel,thesolutionin(10)withthenegativesignistaken,i.e.,8>>0;if^½1=0;>>2½^2>:1¯¯sgn(^½)0:95if¯½^¯¸1:112¯¯Ofcourse,Pr(^½1=0)=0;andif¯½^1¯¸1=2,thismightbeasignalthatanMA(1)modelisnotappropriateforthedata.Anestimateof¾2followsfrom(6)as¾^22Y¾^=;(12)1+^b2where^¾2=S2isthesamplevarianceoftheseriesY;:::;Y.YY1T 141Asecondwayofobtainingstartingvaluesistoevaluatethecondi-tionallikelihoodoverann{lengthgridofb{values,sayb(1);:::;b(n),over(¡1;1),with,from(12),¾2¯xedat^¾2=(1+b2).Y(i)Thatvaluewhichyieldsthelargestlikelihood,sayb,and^¾2=(1+¤Yb2)arethentakenasthestartingvalue.¤AthirdwayistousethefactthatanMA(1)modelcanberep-resentedananin¯niteARmodel,whichsuggestsestimatinganAR(p)model(withplarge)vialeastsquaresandsetting^b=^a1:(13)Observethatifpischosentoosmall,theAR(p)modelisvery"misspeci¯ed,sothat^bwillbequitebiased,whileifpischosentoolarge,thevarianceof^bwillbelarge.Thistradeo®becomesacuteasjbjapproachesone.Thefollowingcodecanbeusedtocomputethisestimatorfortimeseriesy,andchoosesptobeafunctionofthesamplesizewhichiscommoninpractice(KoreishaandPukkila,1990):T=length(y);p=round(sqrt(T));z=y(p+1:end);zl=length(z);Z=[];fori=1:p,Z=[Zy(p-i+1:p-i+zl)];enda=inv(Z'*Z)*Z'*z;b=a(1)Notethattheaccuracyofthesecondmethod,basedonagridsearchoftheconditionallikelihood,willbeafunctionofthegridsize,n,withalargeenoughvalueofnyieldingtheconditionalMLE. 142Ifweaccepttheuseofthetuningparameter"pbeingchosenasthe(roundedup)squarerootofTinthethirdmethod(13),thenitsperformanceandthatofthemoment{basedmethod(11)canbeeasilycomparedviasimulation.Thiswasdoneforthesame10,000seriesusedforthecalculationsshowninFigure26.Figure27showstheresultingbiasandmse.ComparedtoFig-ure26,weseethatthebiasandmseoftheinitialestimatorsareconsiderablylargerthanfortheMLEsasjbjapproachesone,butarecomparableforbnearzero.Method(13)hassmallerbiasthan(11)fortheentireparameterspaceexceptforasmallregionaroundb=¡0:5.Regardingthemse,(11)isslightlybetterthan(13)for¡0:8q.EstimateswhichincorporatethisrestrictionhavebeendevelopedbyWalker(1961). 144MA(q)ProcessesAsjustalludedto,theMA(1)modelcanbeextendedinanaturalwaytotheMA(q)model,givenbyYt=c+Ut+b1Ut¡1+b2Ut¡2+¢¢¢+bqUt¡q=b(L)Ut;whereqb(L)=1+b1L+¢¢¢+bqL:(14)ThemeanofYtisclearlyc.UsingtheiidpropertyoftheUt,Cov(Yt;Yt+s)=E[(Ut+b1Ut¡1¢¢¢+bqUt¡q)(Ut+s+b1Ut+s¡1+¢¢¢+bqUt+s¡q)]q¡jsjX2=¾bibi+jsj;i=0Pq2whereb0´1.Thus,°0=Var(Yt)=i=0biand(P2q¡jsj¾i=0bibi+jsj;jsj·q;°s=Cov(Yt;Yt+s)=(15)0;jsj>q;fromwhichthe½s=°s=°0canbecalculated.Doyouthinkthereisaclosed{form"matrixsolutionavailable? 145(Ourfriend)VanderLeeuw(1994)showsthatthecovariancema-trixofanMA(q)modelwith¾=1canbeexpressedas00§=MM+NN;(16)whereMistheT£Tlowertriangularbandmatrixwith¯rstcolumn(1;b1;:::;bq;0;:::;0)and23"#bqbq¡1¢¢¢b167N160bq¢¢¢b27N=;N1=6.........7;04...500¢¢¢bqwhereNisofsizeT£q.NotethatmatricesMandNparallelthoseofPandQfortheAR(p)case,andthatfortheAR(p)model,§¡1wasdirectlyconstructed,whereasitis§fortheMA(q)model.Forexample,withq=2,¾2=1andT=4,23232310001b1b20b2b1"#6767676b11007601b1b2760b27b2000§=6767+674b2b11054001b154005b1b2000b2b11000100231+b2+b2b+bbb012112262276b1+b1b21+b1+b2b1+b1b2b27=67;4bb+bb1+b2+b2b+bb52112121120bb+bb1+b2+b2211212whichagreeswith(15). 146TheconceptofinvertibilityisalsoextendedtotheMA(q)case.ParallelingthedevelopmentofthestationarityconditionforAR(p)models,expresspolynomial(14)asqb(L)=1+b1L+¢¢¢+bqL=(1¡´1L)¢¢¢¢¢(1¡´qL)(17)sothattheroots6ofb(L)aregivenby´¡1;:::;´¡1.Themodel¯¯1qisinvertiblewhen¯´¡1¯>1or,equivalently,whenj´j<1,i=ii1;:::;q,wherej´ijisthemodulusof´i.UnliketheARmodel,wecan°ip"anysetofthe´iandthecorrelationstructureremainsthesame.Forexample,letb1=¡0:5andb2=¡0:24.As21¡0:5L¡0:24L=(1+0:3L)(1¡0:8L);¯¯therootsare´¡1=¡10=3and´¡1=10=8,sothat,as¯´¡1¯>1,12ii=1;2,themodelisinvertible.Flipping´2givesµ¶12¤¤2¤(1+0:3L)1¡L=1¡0:95L¡0:375L=:1+b1L+b2L=:b(L);0:8whichisobviouslynotinvertible,buttheMA(2)modelsbasedonb(L)andb¤(L)haveexactlythesamecorrelation(butnotcovariance)structure,namely,tofourdigits,½1=¡0:2906,½2=¡0:1835and½i=0,i¸3.Interestingly,theparametersbineednotcorrespondtoaninvert-ibleMAprocessinorderfor(16)tobevalid.6SimilartotheexpressionfortheARpolynomial,ifb=(b;:::;b)istheMAparametervector,then1qexecutingrr=roots([b(end:-1:1)1])computestherootsinMatlab. 147AutoregressiveMovingAverageProcesses(ARMA)ModelsSimilarto(5),considerthein¯niteARmodel2Yt=aYt¡1+aYt¡2+¢¢¢+Ut:Itcanbewrittenas¡¢221¡aL¡aL¡¢¢¢Yt=Utand,withc=aL,thepolynomialis¡¢11¡2c221¡c¡c¡¢¢¢=2¡1+c+c+¢¢¢=2¡=;1¡c1¡csothat1¡2cYt=Ut1¡corYt¡2aYt¡1=Ut¡aUt¡1orYt=2aYt¡1+Ut¡aUt¡1:(18)Model(18)isstationaryforjaj<1=2.ThismodelcombinesanAR(1)withanMA(1)structure,thoughinarestrictedway.RelaxingtheconstraintgivesamodelwiththeformYt=aYt¡1+Ut+bUt¡1;whichis,appropriately,referredtoasanARMA(1,1)model(andisstationaryifjaj<1). 148Moregenerally,bycombiningtheAR(p)andMA(q)structures,theARMA(p;q)modelisobtained:a(L)Yt=c+b(L)Ut;(19)wherepqa(L)=1¡a1L¡¢¢¢¡apLandb(L)=1+b1L+¢¢¢+bqL:ThesameexercisewhichledtothemeanofanAR(p)modelshowsthatE[Yt]=¹=c=(1¡a1¡¢¢¢¡ap):(20)Usingthisvalueof¹,itiseasytoverifythatmodel(19)canbewrittenasa(L)(Yt¡¹)=b(L)Ut:Thus,asinthepureMAcase,themeandoesnotdependontheMAparameters.Forthetimebeing,weconsidertheknownmeancase,sothat,withoutlossofgenerality,let¹=0.Theregressioncase,whichgeneralizestheuseofc,isdealtwithlater. 149ThesimulationofanARMAprocessisquitestraightforward.OnewayistomultiplytheCholeskydecompositionoftheexactvariance{covariancematrixcorrespondingtothespeci¯edARMAmodel(givenbelow)andthenright{multiplyingitbyavectorofiidstandardnormalrandomvariables.ForlargeT,thiswillclearlybetimeconsuming.Instead,wejustusealoopandsetYtequaltotheweightedsumoftheppastvaluesofYandtheqpastvaluesofUdictatedbytheparametersoftheARMA(p;q)model.ImportantisthenthechoiceofstartingvaluesY1¡p;:::;Y0,whichcouldbedeterminedfromtheaforementionedmethodbasedontheexactcovariancematrix.Thoughthatisarguablythebestway,wejustsetvaluesY1¡p;:::;Y0tozero,simulate500+Tobservations,anddeliverthe¯nalTvalues.functiony=armasim(nobs,sig2,pv,qv,seed)ifnargin<5,seed=100*rand;endrandn('state',seed)ifnargin<4,qv=[];endp=length(pv);q=length(qv);pv=reshape(pv,1,p);ifq>0,qv=reshape(qv,1,q);endwarmup=500;e=sqrt(sig2)*randn(nobs+warmup,1);init=0;evec=zeros(q,1);yvec=zeros(p,1);y=zeros(nobs+warmup,1);fori=1:nobs+warmupifp>0,y(i)=y(i)+pv*yvec;endifq>0,y(i)=y(i)+qv*evec;endy(i)=y(i)+e(i);ifp>1,yvec(2:p)=yvec(1:p-1);end,yvec(1)=y(i);ifq>1,evec(2:q)=evec(1:q-1);end,evec(1)=e(i);endy=y(warmup+1:end);ProgramListing0.1:SimulatesanARMAprocesswithinnovationvariancesig2,ARparameterspassedasvectorpvandMAparametersqv 150RecallingthenotationfortheAR(p)modelandtheMA(q)modelQpfrom(17),wecanwritea(L)=i=1(1¡¸iL)and,similarlyforQqtheMApolynomial,b(L)=j=1(1¡´jL),sothat(1¡´1L)(1¡´2L)¢¢¢(1¡´qL)Yt=Ut:(21)(1¡¸1L)(1¡¸2L)¢¢¢(1¡¸pL)Ifthereisapair(i;j)suchthat´i=¸j,thenthetwofactors(1¡¸jL)and(1¡´iL)cancelin(21),andthemodelisexpress-ibleasanARMA(p¡1;q¡1).Moregenerally,anyARMA(p;q)modelisequivalenttoanARMA(p+k;q+k)simplybyintroducingthefactor(1¡g1L)¢¢¢(1¡gkL)inboththenumeratoranddenominatorintherepresentation(21)foranarbitarysetofvaluesg1;:::;gk,subjecttojgij·1,i=1;:::;k.ExampleTheARMA(2,2)processYt¡1:8Yt¡1+0:8Yt¡2=Ut¡0:7Ut¡1¡0:3Ut¡2(22)canbewrittenas(1+0:3L)(1¡L)(1+0:3L)Yt=Ut=Ut;(1¡0:8L)(1¡L)(1¡0:8L)sothatitisequivalenttotheARMA(1,1)processYt¡0:8Yt¡1=Ut+0:3Ut¡1:Simulating(22)with5,000observationsindeedveri¯esthattheprocessisstationary,asshowninthetopleftpanelofFigure28(thetoprightpanelisjustamagni¯edviewofpartoftheseries). 151Itisinstructivetoseewhathappenswhenchangingb1=¡0:7tob1=¡0:69.Thisprocessnolongerexhibitscancellationofterms,andisthusanonstationaryARMA(2,2)model(butis,ofcourse,closetobeingstationary).ArealizationoftheseriesbasedonthesameinnovationsequenceusedforthepreviouscaseisshowninthemiddlepanelsofFigure28.Basedonly"onthe400observationsfromtherightpanel,thereisverylittledi®erencecomparedtoitsstationarycounterpartinthetoprightpanel.Continuingthisexercise,thebottompanelscorrespondtothesameinnovationsequence,butwithb1=¡0:65.Theentireseriesre-semblesathicklydrawn"randomwalk,whilethesegmentintherightpanellooksjustlikethestationarydatashownaboveit,justwithapositivelineartrend. 15285643422100−2−1−4−2−6−3−8−4010002000300040005000240025002600270028001038261402−10−2−2−3−4−4−6−5−8−6−10−70100020003000400050002400250026002700280015−410−65−80−10−5−12−10−14−16−15−18−2001000200030004000500024002500260027002800Figure28:TopisrealizationoftheARMA(2,2)processYt¡1:8Yt¡1+0:8Yt¡2=Ut¡0:7Ut¡1¡0:3Ut¡2asin(22),whichisequivalenttothestationaryandinvertibleARMA(1,1)processYt¡0:8Yt¡1=Ut+0:3Ut¡1.Rightsideshowsmagni¯edviewofpartoftheseries.Middlepanelsshowtherealizationofthealteredmodelwithb1=¡0:69insteadofb1=¡0:7,basedonthesameinnovationsequence.Bottompanelsaresimilar,butwithb1=¡0:65. 153ExampleSimilartothepreviousexample,considerthemodel(1+0:3L)(1¡1:1L)Yt=Ut;(23)(1¡0:8L)(1¡1:1L)i.e.,Yt¡1:9Yt¡1+0:88Yt¡2=Ut¡0:8Ut¡1¡0:33Ut¡2.Itisnotstationary;simulatingitresultsinanexplosiveprocess.Thereasonthattheterms(1¡1:1L)donotcancelisthatthedynamicsinducedbytheMApolynomialareequivalenttothoseformedbyinvertinganysetofroots.Inparticular,theprocessdescribedin(23)hasthesamepropertiesas(1+0:3L)(1¡1=(1:1)L)Yt=Ut;(1¡0:8L)(1¡1:1L)andcancellationoftheo®endingtermsisnolongerpossible.Whenthereisanidenticalterm,(1¡gL),jgj·1,inboththenumeratoranddenominatorofrepresentation(21),themodelissaidtobesubjecttozeropolecancellation,atermusedmainlyintheengineeringliterature.Itshouldbeobviousthattheparametergisnotidenti¯ed,sothatestimatinganARMA(p+k;q+k)model,k¸1,basedondatageneratedfromanARMA(p;q)processisnotpossible.Ofcourse,inpractice,onehasa¯niteamountofdata,sothat,whenestimatingwithmaximumlikelihood,theremightbealocalmaximumwhichthenumericproceduresettlesupon.However,alltheARMAparameterswillhaveverylargestandarderrors,andcomputationoftherootsoftheARandMApolynomialswillrevealthatkrootsare(approximately)shared. 154In¯niteARandMARepresentationsWehavealreadyseenfrom(8)thataninvertibleMA(1)modelcanberepresentedbyanin¯niteARmodel.Similarly,astationaryAR(1)modelcanbeexpressedasanin¯niteMA,i.e.,ifYt=aYt¡1+Ut,or(1¡aL)Yt=Ut,jaj<1,then¡1222Yt=(1¡aL)Ut=(1+aL+aL+¢¢¢)Ut=Ut+aUt¡1+aUt¡2+¢¢¢:Theseresultsgeneralize:AninvertibleMA(q)processcanberepre-sentedasanin¯niteAR,andastationaryAR(p)canberepresentedasanin¯niteMA.Toillustratethelatter,considerthestationaryAR(p)processa(L)Y=Uwitha(L)=1¡aL¡¢¢¢¡aLp.Thein¯niteMAtt1prepresentationisgivenby¡¢¡12Yt=a(L)Ut=Ã(L)Ut=1+Ã1L+Ã2L+¢¢¢Ut:Thecoe±cientsinÃ(L)canbeobtainedbytreatingLasthevariableinapolynomialandmultiplyingbothsidesofa¡1(L)=Ã(L)bya(L),i.e.,1=a(L)Ã(L),andthenequatingcoe±cientsofLj.Thatis,¡¢p2a(L)Ã(L)=(1¡a1L¡¢¢¢¡apL)1+Ã1L+Ã2L+¢¢¢2=1+(Ã1L¡a1L)+(Ã2¡a1Ã1¡a2)L3+(Ã3¡a1Ã2¡a2Ã1¡a3)L+¢¢¢sothatÃ1¡a1=0)Ã1=a1Ã2¡a1Ã1¡a2=0)Ã2=a1Ã1+a2Ã3¡a1Ã2¡a2Ã1¡a3=0)Ã3=a1Ã2+a2Ã1+a3... 155orXjÃ0=1;Ãj=akÃj¡k;j=1;2;::::k=1Notethat,foranAR(1)process,a=0fori¸2,andÃ=ai,ii1i=0;1;2;:::.Besidesbeingoftheoreticalinterestandalsorequiredforforecast-ing(seebelow),thein¯niteMArepresentationshowsthatARandMAmodelsareonlyinterchangeable"whenin¯nitenumbersoftermsareallowedfor.If,forexample,wewererestrictedtousinganAR(p)modelto¯tdatageneratedfromanMA(1)process,wewouldneedtotakepverylarge(ideallyin¯nity)forforecastingtheseries.Assuch,therestrictiontopureARorpureMAprocessesmightbetoolimitedforcapturingthecorrelationstructureobservedfromagivendataset.Bycombiningthetwostructures,very°exiblecorrelationsequences½1;½2;:::canbeapproximatedwithfarfewerparametersthanwouldberequiredwithpureARorMAmodels.Thisisthenatureofparsimoniousmodelbuilding:usingasfewparametersaspossibletocapturethesalient,ormostpromi-nentfeaturesofthedata.(Wetalkaboutselectingpandqforaparticulartimeserieslater.) 156Thepreviousderivationofthein¯niteMAexpressionforanAR(p)modelcanbeextendedtothemixed,i.e.,(stationaryandinvert-ible)ARMA(p;q)caseinastraightforwardway:ThemodelisnowY=a¡1(L)b(L)U=Ã(L)U,sothatweequatecoe±cientsoftttLjinb(L)=a(L)Ã(L)togettheÃi.WithÃ0=1,itisstraightforwardtoverifythatthisPjleadstotherecursiveexpressionÃj=bj+k=1akÃj¡kor,usingthe¯nitenessofpandq,min(j;p)XÃ0=1;Ãj=bjI(j·q)+akÃj¡k;j¸1:(24)k=1Similarly,wecanexpressthemodelas¼(L)Y=b¡1(L)a(L)Y=ttUt,where2¼(L)=1¡¼1L¡¼2L+¢¢¢isthein¯niteARpolynomial,thetermsofwhicharecomputedbyequatingcoe±cientsofLjinb(L)¼(L)=a(L)togetmin(j;q)X¼0=¡1;¼j=ajI(j·p)¡bk¼j¡k;j¸1:(25)k=1Thein¯niteARrepresentationisthenYt=¼1Yt¡1+¼2Yt¡2+¢¢¢:(26) 157SomeBonusMaterial...Ifwezero{padtheARandMApolynomials,i.e.,setai=0ifi>pandbi=0ifi>q,then(25)is¼j=aj¡b1¼j¡1¡¢¢¢¡bj¡1¼1+bjor,asiseasilyveri¯ed,~a=(Ir+1+B)¼;(27)where232323¡1¡100¢¢¢006767676¼176a176b100076677667766......77¼=6¼27;~a=6a27andB=6b2b1...7:6..76..76....74.54.54..005¼rarbrbr¡1¢¢¢b10¡1Thiscouldbeusedtogetthematrixexpression¼=(Ir+1+B)~a,where¡1(Ir+1+B)alwaysexistsbecausejIr+1+Bj=1.Thiscouldbecomputedforr=pandthen,forj>p,therecursion¼j=¡b1¼j¡1¡b2¼j¡2¡¢¢¢¡bq¼j¡qwouldbeused.Anothervalueofthisexerciseisthatthelatterequationbeusedtoexpressthebiintermsofthe¼ibybuildingasystemofequationsforj=p+1;:::;p+q,i.e.,with¼0=1and¼i=0fori<0,232323¼p+1¼p¼p¡1¢¢¢¼p+1¡qb16767676¼p+276¼p+1¼p¢¢¢¼p+2¡q76b276..7=¡6....76..7=:¡¦b;(28)4.54..54.5¼p+q¼p+q¡1¼p+q¡2¢¢¢¼pbq0whichcanbesolvedinanobviouswayforb=(b1;:::;bq),providedmatrix¦isoffullrank.Withbknown,(27)canbecomputedwithr=ptoobtain0~aand,thus,a=(a1;:::;ap).Inotherwords,theparametervectorsaandbcanbeeasilyrecoveredfrom¼1;:::;¼p+q. 158Likelihood{BasedEstimationTheexactMLEoftheparametersofanARMAmodelcanbe(numerically)obtainedonceacomputableexpressionisavailablefor§.Recallingthede¯nitionsofmatricesPandQintheAR(p)case,andMandNin(16),vanderLeeuw(1994)showsthatthecovariancematrixofaT{lengthtimeseriesgeneratedbyastationaryandinvertibleARMA(p;q)processcanbeexpressedas£P¹0P¹¡Q¹Q¹0¤¡10§=[NM][NM];(29)whereP¹andQ¹havethesamestructureasmatricesPandQ,butareoforder(T+p)£(T+p)and(T+p)£p,respectively.7ExampleFortheARMA(1,1)modelYt=aYt¡1+Ut+bUt¡1withT=2,2323100a"#"#P¹=6¡a10760710;N=b45;Q¹=45;M=b100¡a10and(29)gives23¡123"#1¡a0b0"#b1062767°0°1§=4¡aa+1¡a541b5=;0b1°1°00¡a101where1+2ab+b2(1+ab)(a+b)°0=;°1=1¡a21¡a2andhigherordercovariancesarecomputedas°i=a°i¡1,i¸2.7As[NM]isT£(T+q),butP¹0P¹andQ¹Q¹0areofsize(T+p)£(T+p),(29)isclearlyproblematicforp6=q.Tosolvethis,oneneedstozero{padthea(L)orb(L)tohavelengthm=max(p;q).VanderLeeuwdoesnotexplicitlystatethis,butthepenultimatesentenceinhisx4alludestoit. 159Whilebothelegantandeasilyprogrammed,(29)willbeexceed-inglyslowasTgrows.Thisproblemcanbeeliminatedbycalculat-ing(29)forT¤=m+1,m=max(p;q),andusingthefollowingrecursionfortheremainingelements:Xp°k=ai°k¡i;k=m+2;:::;T:(30)i=1Toseethevalidityof(30),¯rstzero{padtheARorMApolynomialsothatp=q=m,thenmultiplytheequationforYtbyYt¡k(assumingE[Yt]=0withoutlossofgenerality),YtYt¡k=a1Yt¡kYt¡1+¢¢¢+amYt¡kYt¡m+Yt¡kUt+b1Yt¡kUt¡1+¢¢¢+bmYt¡kUt¡m;andtakeexpectationstoget(usingthefactthat°i=°¡i)Xm°k=a1°k¡1+¢¢¢+am°k¡m+E[Yt¡kUt¡i]:(31)i=0AsE[Yt¡kUt¡i]=0ift¡i>t¡k,ork>i,thelattersumin(31)iszeroifk>m,whichjusti¯es(30).Theonlyreasonkstartsfromm+2insteadofm+1in(30)isthat(29)requiresT>m.The¯rstexplicitcomputer{programmablemethodforcalculating°m=(°0;:::;°m)foranARMAmodelappearstobegivenbyMcLeod(1975)andTunnicli®e{Wilson(1979),although,asMcLeodalsostates,themethodwasusedforsomespecialARMAcasesinthe¯rstedition(1970)oftheseminalBoxandJenkinsmonograph.Aclosed{formmatrixexpressionfor°mappearstohavebeen¯rstgivenbyMittnik(1988),whileZinde{Walsh(1988)andKaranasos(1998)deriveexpressionsfor°ibasedonthebiandtherootsoftheARpolynomial,withKaranasos'resultrestrictedtothecasewithdistinct(realorcomplex)roots. 160Once§isnumericallyavailable,thelikelihoodisstraightforward(inprinciple)tocalculateandmaximize.Thedrawback,however,ofanymethodforcalculating§,whateveritsspeed,isthataT£Tmatrixinverseneedstobecalculatedateachlikelihoodevaluation.KeepinmindthatthisproblemevaporateswhenworkingwithpureAR(p)models:theexactlikelihoodispartitionedsothatonly§¡1ofsizep+1needstobecalculated|and§¡1canbedirectlycalculatedviavanderLeeuw'sformula,thusevenavoidingthesmallmatrixinversion.WithMAorARMAprocesses,thisluxuryisnolongeravailable.AsTgetsintothehundreds,thecalculationof§¡1forMAorARMAprocessesnotonlybecomesprohibitivetime{wise,butalsonumericalround{o®errorinthematrixinversioncouldleadtoinaccuraciesorconvergenceproblems.ThemethodinvolvingthestatespacerepresentationmentionedearlieranduseoftheKalman¯lterismuchpreferredforcomputingtheexactMLE.Alternatively,theconditionalMLEcanbecomputed,whichsimplycombinestheconditioningargumentsusedintheseparateARandMAcases.Inparticular,the¯rstpvaluesofYtareassumed¯xed,andallqunobservablevaluesofUtaretakentobezero.Theconditionallikelihoodstillneedstobenumericallymaximized,butastherearenoT£Tmatricestoinvert,themethodisenormouslyfasterforlargeTand,unlesstheARand/orMApolynomialsareclosetothestationarity(invertibility)borders,therewillnotbemuchdi®erenceintheconditionalandexactMLEvalues. 161SimilartothedevelopmentfortheAR(1)model,wecanintro-ducearegressiontermintotheARMAmodelviatheobservationequation0Yt=xt¯+²t;(32)butwiththelatentequationbeinggivenbytheARMAprocessa(L)²t=b(L)Ut.0ObservethatthejointdistributionofY=(Y1;Y2;:::;YT)is¡¢X¯;¾2§,whereX=[x;:::;x]0andisassumedtobefullN1Trank,ofsizeT£k,and¾2§istheT£Tcovariancematrixof0Y¡X¯=(²1;²2;:::;²T).As§isreadilycomputablevia(29)and(30),theexactlikelihoodofYcanbestraightforwardlycomputedand,thus,theMLEof¡¢0theparametervector¯0a0b0¾canbeobtained,wherea=(a1;:::;ap)andb=(b1;:::;bq).IfXisjustacolumnofones,i.e.,(19),butthewayofintroducingtheconstanttermintothemodelisdi®erent.Inparticular,with(19),themeanisgivenby(20),whereaswith(32),themeanis¯1. 162IntervalEstimationAsymptotically,undercertainassumptions,forboththeexactandconditionalMLEofµ=(a;:::;a;b;:::;b)0,1p1q"#¡1p³´asyE[XX0]E[XV0]Tµ^¡µ»N(0;C);C=¾2;ML00E[VX]E[VV](33)whereX=(Xt;Xt¡1;:::;Xt¡p+1),witha(L)Xt=UtandV=(V;V;:::;V)0withb(L)V=U.Thisclearlygeneralizestt¡1t¡q+1tttheAR(p)andMA(q)results.Forproof,seethereferencesgivenearlier.ExampleForp=q=1andusingthein¯niteMArepresenta-tionsfortheARpolynomials,2Ã!013X1X1E[XV]=E4aiU@(¡b)jUA5ttt¡it¡ji=0j=0X11i=(¡ab)=1+abi=0sothat"22#¡1¾¾C=¾21¡a21+ab¾2¾21+ab"1¡¡b2#¢¡¢¡¢1+ab1¡a2(1+ab)¡1¡a21¡b2=¡¢¡¢¡¢:2¡1¡a21¡b21¡b2(1+ab)(a+b) 163Observethat¡¢22limlimVar(^aML)=1¡a=ab!0T!1and³´limlimVar^b=1=a2MLb!0T!1sothatifanAR(1)processiswronglyestimatedwithanARMA(1;1)model,thevarianceof^aMLand^bMLincreasewithoutboundasaapproacheszero.Thatis,inthelimit,therewillbezeropolecan-cellationandtheparametersarenolongeridenti¯ed.Thiscanalsobeseenanotherway:ForanAR(1)modelwithacloseto0,thein¯niteMArepresentationisÃ=a,Ã=ai¼0,1ii¸2,whichisvirtuallythesameasthatofanMA(1)processwithb=a.Thus,fora¼0,thetwoARMAparametersofanARMA(1;1)modelarenotidenti¯ableandtheirjointestimationbecomesimpossible.Thisnonidenti¯abilityexpressesitselfin¯-nitesamplesbyabnormallylargestandarderrorsfortherelevantparameters.GodolphinandUnwin(1983)havedevelopedane±cientalgorithmtoal-leviatetheotherwisetediousevaluationofmatrixCin(33)whenp+qisnotverysmall.Withit,CcanbenumericallycomputedwiththeMLEvaluesµ^MLreplacingµ.Potentiallyeasier(andpossiblymoreaccurate)istousetheapproximateHessianmatrixfromthelikelihoodfunction.Basedonitandtheasymptoticnormalityoftheestimators,approximateone{at{a{timecon¯denceintervals(CIs)foreachoftheparameterscanbeconstructed. 164UseoftheBootstrapAwayofobtainingmoreaccurateintervalestimatorsforthepa-rametersistousethebootstrap.Inparticular,giventheestimatedresidualsU=(U^;:::;U^)01T(whichareapproximatelyiidnormalifthetruedatageneratingprocessisastationaryARMA(p;q)modelwithnormalinnova-tionsandpandqarecorrectlyspeci¯ed),letU(i)betheithboot-strapreplicationoftheU,formedbysamplingfromtheU^twithreplacement.ForeachU(i),generatetimeseriesY(i)=§1=2U(i),where§=§(µ^ML)isbasedontheMLEoftheoriginaldata,andcomputethe(i)correspondingMLEµ^ML.ThisisconductedBtimesandtheusualmethodofobtainingcon¯denceintervalsforeachoftheparametersisused.ExampleForp=q=1,a=¡0:3,b=0:7andusingB=2000bootstrapreplications,90%CIswereconstructedforaandbfor1,000simulatedtimeserieswithT=30and¾2=16.Thelengthofeachintervalandwhetherornotitcoveredthetrueparameterwererecorded.ThiswasalsodoneforintervalsbasedontheapproximateHessianmatrixreturnedwiththeMLEvaluesandtheasymptoticnormaldistribution.Forparametera,theactualcoverageoftheasymptoticCIwas0.77,withaveragelength0.96.ThebootstrapCIhadactualcoverage0.91andmeanlength1.3.Thus,forthisrelativelysmallsamplesize,thebootstrapintervalappearsfarsuperiortotheuseoftheasymptoticresult.Forparameterbhowever,coverageoftheasymptoticCIwas0.72withlength0.96,whilethebootstrapcoveragewas0.98withlength1.5.WhilestillfarcloserincoveragethantheasymptoticCI,thebootstrapintervalisapparentlytoolarge. 165ForecastingARMA(p;q)ModelsIt'stoughtomakepredictions,especiallyaboutthefuture.(YogiBerra)Let'sbeginwiththeAR(p)process.Forpointprediction,theextensionoftheAR(1)casewedevelopedearliertotheAR(p)caseisstraightforward:Forh=1,pointforecastY^T+1jTisformedbysubstitutingestimatesinplaceofunknownsintotherhsofXpYT+1=aiYT+1¡i+UT+1;i=1sothatUT+1isreplacedbyzeroandaiisreplacedby^ai.Fortheone{stepaheadforecastmseofanestimatedAR(p)model,let00a=(a1;a2;:::;ap);^a=(^a1;a^2;:::;a^p)and0Yt=(Yt;Yt¡1;:::;Yt¡p+1):Then¡¢h¡¢iY^Y^2mseT+1jT=ET+1jT¡YT+1h¡¢i=EY^¡a0Y+a0Y¡Y2T+1jTTTT+1h¡¢ihi=EY^¡a0Y2+E(a0Y¡Y)2+crosstermT+1jTTTT+1hi=E((^a0¡a0)Y)2+¾2:T 166Letdi=a^i¡ai.ThenhihiE((^a0¡a0)Y)2=E(dY+dY+¢¢¢+dY+¢¢¢+dY)2T1T2T¡1iT¡i+1pT¡p+1XpXp=E[diYT¡i+1djYT¡j+1]:i=1j=1Then,pretending"thatthediareindependentoftheYT(theyarenotinsmallsamples,butareasymptotically),wecanwritehiXpXp002E((^a¡a)YT)¼Cov(^ai;a^j)°j¡ii=1j=1µ¶¾2¾2¡¢0¡10¡1=1¡¯¡1=1¡¯¡1TT¾2=p;Twhichfollowsbecause,foranyfullranksymmetricmatrixMof¡¢sizem,10M¯M¡11=m,where¯denotestheHadamard(orelementwise)matrixproduct.Thus,weobtainthepleasantlysimpleexpression¡¢³p´mseY^¼¾21+:(34)T+1jTTThisappearstohavebeen¯rstgivenbyBloom¯eld(1972,p.505),derivedinthecontextofthespectralanalysisoftimeseries.Theeasiestwayofdeterminingthequalityofapproximation(34)isviasimulation.Youarenotresponsibleforthederivationof(34),butyouneedtoknowtheformula. 167FortheAR(2)modelwitha=1:2,a=¡0:8and¾2=4,12¡¢thetruemseY^T+1jTbasedontheexactMLEand100,000repli-cationsis4.99,4.44and4.30forT=10;20and30,respectively,whichcanbecomparedtothevaluesfrom(34)of4.8,4.4and4.27.Similarly,fora1=¡0:6,a2=0:2,simulationresultedin4.91,4.45and4.29forT=10;20and30.Thoughfurthersimulationwouldberequiredbeforemakinggeneralstatements,itappearsthat(34)isalmostexactforT=30andisstillvastlybetterfor10p:Ofcourse,Y^T+hjTcanbeexpressedasalinearcombinationofYT;YT¡1;:::;YT¡p+1;fortheAR(3)casewithh=2,Y^T+2jT=^a1Y^T+1jT+^a2YT+^a3YT¡1=^a1(^a1YT+^a2YT¡1+^a3YT¡2)+^a2YT+^a3YT¡1¡¢2=a^1+^a2YT+(^a1a^2+^a3)YT¡1+^a1a^3YT¡2:(35)(2)Thecoe±cientofYT¡icanbedesignatedas^aiand,ingeneral,Y^Pp(h)T+hjT=i=1a^iYT¡i+1. 169(h)Asimplewayofcalculatingthe^aforanyhandpistoexpressithemodelasthevectorAR(1)processYt=AYt¡1+Ut;00whereYt=(Yt;Yt¡1;:::;Yt¡p+1),Ut=(Ut;0;:::;0)and23a1a2¢¢¢ap¡1ap6761000767A=66010077:6...............74500¢¢¢10WithE[U]=0forh¸1,Y=AhYsothattheT+hT+hjTT(h)ha^iaretheelementsinthe¯rstrowofA^andY^T+hjTisthe¯rsth0elementofY^T+hjT=A^YT.Notethat,withe1=(1;0;:::;0),Y^=e0A^hY.Forexample,thecoe±cientsin(35)aregivenT+hjT1Tbythe¯rstrowin232323a^a^a^a^a^a^a^2+^aa^a^+^aa^a^1231231212313A^2=61007610076a^a^a^74545=41235:010010100Also,simplesubstitutionasinthescalarAR(1)caseshowsthatXh¡1hiYT+h=AYT+AUT+h¡i;(36)i=0andY=e0Y.Forp=1,the¯rsttermin(36)isjustahYT+h1T+h1Tand,ifjaj<1,thenah!0ash!1.11Presumably,iftheAR(p)modelisstationary,thenAh!0ash!1. 1700Thisconjectureistrue,andseenbywritingA=U¤U,whereUisorthogonaland¤=diag(¸1;:::;¸p)aretheeigenvaluesofA.Then,Ah!0ifeachj¸j<1.Itisstraightforwardtoshowi(see,e.g.,Hamilton,1994,p.21)thatthe¸iarethevalueswhichsatisfy¡11¡p¡p1¡a1¸¡¢¢¢¡ap¡1¸¡ap¸=0;or,recallingthepropertyofstationarityAR(p)processes,thatstationarityoftheARprocessisequivalenttoAh!0.PThen,(36)impliesthatY=1AiU,i.e.,thate0Aie=Ã,ti=0t¡i11ii.e.,theithterminthein¯nitemovingaverageexpressionforY.¡¢tThus,mseY^T+hjTish¡¢iEY^¡e0AhY+e0AhY¡Y2T+hjT1T1TT+h"#h¡¢i¡¡Xh¡1¢¢=Ee0A^hY¡e0AhY2+Ee0AhY¡e0AhY+AiU2+01T1T1T1TT+h¡ii=0"#h¡¡¢¢i¡Xh¡1¢=Ee0A^h¡AhY2+E¡e0AiU21T1T+h¡ii=0Pandthelattertermisjust¾2h¡1Ã2.i=0iTheformertermissigni¯cantlymoredi±culttohandleforgeneralhthanitwaswithh=1becausetheasymptoticcovariancematrixofAhwillinvolveitsderivativewithrespecttoa.Therequiredma-trixdi®erentiationresultwasderivedinNeudecker(1969),which¡¢Yamamoto(1976)usedtogiveanexpressionformseY^T+hjT.Assuch,wecontentourselveswiththeexpression¡¢Xh¡1mseY^¼¾2Ã2(37)T+hjTii=0¡¢whichisclearlyanunderestimateofmseY^T+hjTbecauseitne-glectstheforecasterrorarisingfromtheparameteruncertainty,althoughitisasymptoticallycorrect. 171MattersincreaseincomplexitywhendealingwithARMA(p;q)processes,compoundedevenfurtherwithanunknownmeanterm.WenowturntotheMAandARMAcases.FortheMA(q)model,thesameprinciplesasaboveareapplied.Forpointestimates,Y^T+hjT=^b1U^T+h¡1+^b2U^T+h¡2+^b3U^T+h¡3+¢¢¢+^bqU^T+h¡qXq=^bU^=b^0U^;jT+h¡jT+h¡1j=1¡¢0whereU^t=U^t;U^t¡1;:::;U^t¡q+1.AsnoneoftheU^tareob-served,the¯lteredvalues,i.e.,themodelresidualsU^t,t=1;:::;T,areusedintheirplace.Ift>T,thenE[Ut]=0isused.Forexample,withq=2,Y^T+1jT=^b1U^T+^b2U^T¡1;Y^T+2jT=0+^b2U^T;andY^T+hjT=0;h>q:Someworkshowsthat·µXh¡1¶2¸¡¢mseY^T+1jT¼E¡bjUT+h¡jj=0¡¢2222=¾1+b1+b2+¢¢¢+bh¡1;(38)wherebj=0ifj>q.Observethatthisisthesameas(37),i.e.,wheretheÃiarethe(inthiscase¯nite)termsinthepuremovingaveragerepresentationofthemodel.Itisimportanttokeepinmindthat,inboth(37)and(38),parameteruncertaintyisnottakenintoaccount. 172PointestimatesfortheARMA(p;q)casefollowsanalogouslybycombiningthetechniquesintheARandMAspecialcases.Forexample,withp=3andq=2,Y^T+1jT=^a1YT+^a2YT¡1+^a3YT¡2+^b1U^T+^b2U^T¡1;Y^T+2jT=^a1Y^T+1jT+^a2YT+^a3YT¡1+0+^b2U^T;Y^T+3jT=^a1Y^T+2jT+^a2Y^T+1jT+^a3YT;andY^T+hjT=^a1Y^T+h¡1jT+^a2Y^T+h¡2jT+^a3Y^T+h¡3jT;h>max(p;q):Ingeneral,usingthepreviousde¯nitionsofa,b,YTandUT,Y=a0Y+b0U^,Y=e0AhY+b0U^andT+1jTTTT+hjT1TT+h¡1Y^=e0A^hY+b^0U^wherethe¯lteredresidualsU^areT+hjT1TT+h¡1tusedfort=1;:::;Tandzeroift>T.Inlightof(37)and(38),onemightexpectfortheARMAcasethat³´Xh¡1mseY^¼¾2Ã2:(39)T+hjTii=0whenparameteruncertaintyisignored.Thisisindeedthecaseandcanbederivedalongsimilarlinesasbefore.Ananalysisofthe(downward)biasinherentin(39)isanalyzedindetailbyAnsleyandNewbold(1981).They¯nd,amongotherthings,thatthebiasismoreextremethecloserthemodelistothestationarityand/ortheinvertibilityborder.ThebootstrapcouldbeusedtocomputemoreaccurateCIsforthepointforecasts.Itwouldbeusedjustasdescribedearlier,butinsteadof(orinadditionto)keepingthebootstrapparameteres-timates,foreachresampledseries,anh{stepaheadforecastwouldbemade,butwiththeUT+ichosenfromtheU^twithreplacementinsteadofsettozero. 173ExampleAsampleseriesoflengthT=100basedontheARMA(2;1)modelwitha2=1:2,a2=¡0:8b1=0:5,¾=1andinterceptandtrendregressorterm10+0:04twassimulated,itsparametersestimatedandthe¯rst30out{of{sampleforecastsconstructed,basedonboththetrueandestimatedparameters.TheexactMLEs(withstandarderrorsinparentheses)were¯^1=10:71(0:48),¯^2=0:0278(0:0081),^a1=1:125(0:075),^a2=¡0:747(0:072),^b1=0:451(0:11)and^¾=0:983(0:070).TheseriesandtheforecastsareshowninFigure29,alongwith90%CIsbasedonuseof(39)withtheestimatedvalues,i.e.,P¾^2h¡1Ã^2.(Theestimatedregressionlinewasaddedtotheloweri=0ianduppervaluesfortheARMACIs.)Weseethataconsiderableportionofthedi®erencebetweenthepointforecastsbasedontheestimatedandtrueparameterscanbeattributedtothetrendline,towhichtheyconvergequickly.Also,afteraboutthe5{step{aheadforecast,thesizeoftheCIsareapproximatelythatofthe(detrended)timeseriesitself,renderingaccurateforecastsmuchfurtherthan,say,twostepsaheadalmostimpossible.ThesizeoftheCIfortheone{step{aheadforecastishowevercon-siderablysmallerthanthatforthedetrendedthetimeseriesitself,butrecallthatitdoesnottakeparameteruncertaintyintoaccount. 174Finally,theconditionalMLEwascomputed;thelargestrelativepercentagedi®erencefromtheexactMLEwasfor¯2,whichchangedto0.0269(justover¡3%);theotherparameterschangedbylessthanonepercent.IfoneweretooverlaytheforecastsbasedontheconditionalMLEontotheplot,theywouldbevirtuallyindistin-guishablefromthepointforecastsbasedontheexactMLE.Thedi®erencebetweentheirpointforecastsisthus,relativetothedi®erencebetweenpointforecastsbasedontheestimatedandtrueparameters,essentiallyzero.Thisminutedi®erencebecomescompletelynegligiblewhenconsideringthewidthoftheCIs,i.e.,takingtheuncertaintyofthefutureUtintoaccount.Assuch,iftheprimarygoaloftheanalysisisforecasting,thenuseoftheconditionalMLEinsteadoftheexactMLEappearsaccept-able(andessentiallyobviatestheneedforthemoresophisticatedestimationtechniquesdiscussedabove).Addingabitofinsulttoinjurytotheanalysisinthepreviousexam-ple,oneshouldkeepinmindthattheexerciseusedtheknowledgethatthetruedatageneratingprocessisacovariancestationaryARMA(2,1)modelwithnormalinnovations.Forrealtimeseries,notonlywillpandqnotbeknown,buttheARMAclassitselfmaynotbeappropriate,evenifthedataarecovariancestationary,whichcanoftentimesbeanunrealisticassumption. 175Also,theassumptionthattheinnovationsequenceis(iid)normalmaynotbetenable,withthemostcommondeviationsfromnor-malitybeingfattertailsandskewness.ThispointwasrecentlyinvestigatedindetailbyHarveyandNew-bold(2003)8usingforecasterrorsbasedonmacroeconomictimeseries.Theyconclude...thatthefrequentlymadeassumptionofforecasterrornormalityisuntenable,itsuseresultinginoverlynarrowpredictionintervals"andthat...evidenceofskewnesswasalsodisplayedforthevastmajorityofvariablesandhorizons".Theyrecommendreplacingthenormalassumptionwithaskewedt,suchasthenoncentralt.ObservethatexactMLestimationwithsuchadistributionalassumptionisnotstraightforward,lendingevenmoresupportforuseoftheconditionalMLE.18161412108620406080100120Figure29:Simulatedtimeseries(solid)without{of{sampleforecastsbasedonestimatedparameters(circles)andbasedonthetrueparameters(crosses).Straightdashed(dash{dot)lineistheestimated(exact)regressionterm.8The¯rstauthorisDavidI.Harvey,notthe(moreproli¯c)AndrewC.Harvey. 176TheoreticalandSamplePartialAutocorrelationFunctionRecallthatthesthelementoftheACF,½,measuresthecorrelationsbetweenYtandYt¡s.Themeasureisunconditionalinthesensethatitdoesnotconditiononanyrandomvariables,inparticular,thoselyingbetweenYtandYt¡s.TakethezeromeanAR(1)modelforexample,withYt=aYt¡1+Ut:ItisclearfromtheconstructionoftheYtthat,ifa6=0,thenY1andY3willnotbeindependent;theyarejointlynormallydistributedwithcorrelationa2.If,however,weconditiononY2,thentheconditionalcorrelationbetweenY1andY3willindicatetheir(linear)associationoverandabovetheassociationresultingfromtheirmutualrelationshipwithY2.Thisisreferredtoasthepartialautocorrelationatlags=2.Ititequivalenttotheexpressionofthepartialcorrelationinouranalysisofthemultivariatenormaldistribution,with½ij=Corr(Yt;Yt¡(j¡i))=Corr(Yi;Yj):StillcontinuingwiththeAR(1)model,½¡½½½¡½½a2¡a2131223211½13j(2)=p22=p22=2=0;(1¡½)(1¡½)(1¡½)(1¡½)1¡a122311because2½13=Corr(Yt;Yt¡2)=½2=aand½12=½23=Corr(Yt;Yt¡1)=½1=a: 177Thatis,YtandYt¡2areconditionallyuncorrelatedafterhavingtakenintoaccounttheircorrelationwithYt¡1.Observehowitwascriticaltoconditionontheobservation(s)be-tweenthetworandomvariablesofinterest.ConditionalonY2,¡¢Y=(aY+U)»NaY;¾2and3232Y4=aY3+U4=a(aY2+U3)+U4¡¢2=aY2+aU3+U4¡¡¢¢222»NaY2;¾a+1:ThecovariancebetweenY3andY4conditionalonY2isthen,frombasicprinciples,¾34j(2)=Cov(Y3;Y4jY2)=E[(Y3¡E[Y3jY2])(Y4¡E[Y4jY2])jY2]£¡¢¤2=E(Y3¡aY2)Y4¡aY2jY2£¤232=EY3Y4¡Y3aY2¡aY4Y2+aY2jY2£¡¢¤2=E(aY2+U3)aY2+aU3+U4jY2£¤£¤232¡EY3aY2jY2¡E[aY4Y2jY2]+EaY2jY232222322=aY2+a¾¡aY2aY2¡aY2aY2+aY2=a¾;sothattheconditionalcorrelationisgivenbyCov(Y3;Y4jY2)½34j(2)=Corr(Y3;Y4jY2)=pVar(Y3jY2)Var(Y4jY2)a¾2a=p=p;¾2¾2(a2+1)1+a2whichisonlyzeroifa=0(inwhichcasealltheobservationsareiid). 178The(theoretical)partialautocorrelationfunctionisgivenby(®11;®22;:::;®mm),wheretypicalelement®ssisde¯nedtobethepartialcorrelationbetweenYtandYt¡sconditionalontheYibetweenthetwo,i.e.,®11=½1and®ss=½t;t¡sj(t¡1;:::;t¡s+1)=½1;1+sj(2;:::;s);s>1:(40)Itcanbeshownthatanequivalentde¯nitionofelement®ssisthelastcoe±cientinalinearprojectionofYtonitsmostrecentsvalues,i.e.,Y^t=®s1Yt¡1+®s2Yt¡2+¢¢¢+®ssYt¡s(41)(Thisde¯nitionexplainstheuseofthedoublesubscripton®.)FortheAR(1)model,thisimpliesthat®11=½1=aand®ss=0,s>1.WhenthisPACFisplottedascorrelogram,itwillhaveonlyone(the¯rst)spike;theothersarezero. 179FortheAR(p)model,thisimpliesthat®ss=0fors>p.Figure30showsthePACFforsomeAR(3)examples.Noticehowthevalueofthelastnonzerospike"isalwaysequaltothevalueofthelastnonzeroautocorrelationcoe±cient.0.60.80.60.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.8−0.60510152025051015202510.80.80.60.60.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.6−0.8−0.8−105101520250510152025Figure30:TPACFofthestationaryAR(3)modelwithparametersa=(a1;a2;a3)=(0:4;¡0:5;¡0:2)(topleft),a=(1:2;¡0:8;0)(topright),a=(¡0:03;0:85;0)(bottomleft)anda=(1:4;¡0:2;¡0:3)(bottomright). 180Itcanalsobeshownthat®sscanbecomputedbysolvingthesystemofequations23232323½1½0½1¢¢¢½s¡1®s1®s166½7766½.........7766®7766®7721s2s26..7=6..76..7=:Rs6..7;4.54.½154.54.5½s½s¡1½s¡2¢¢¢½0®ss®ss(42)andwherethe½icouldbeobtainedfromourformulae(vanderLeeuw)forthecovariancestructure.Infact,becauseonlyvalue®isrequiredfrom(42),Cram¶er'srule9sscanbeused,i.e.,jR¤js®ss=;s=1;2;:::;(43)jRsjwherematrixR¤isobtainedbyreplacingthelastcolumnofmatrixsRbyvector(½;½;:::;½)0,i.e.,s12s231½1¢¢¢½s¡2½1676½11½s¡3½2767¤66½2½1½s¡4½377Rs=6......7:6...7674½s¡21½s¡15½s¡1½s¡2¢¢¢½1½s9AfterafterGabrielCramer,1704{1752,whowasborninGeneva. 181Applying(43),the¯rstthreetermsofthePACFaregivenby¯¯¯¯¯1½1¯¯¯j½j¯½1½2¯½¡½21¯¯21®11==½1;®22=¯¯=2;(44)j1j1¡½¯1½1¯1¯¯¯½11¯and¯¯¯¯¯1½1½1¯¯¯¯½11½2¯¯¯¯½2½1½3¯½+½½(½¡2)¡½2(½¡½)¯¯3122131®33==:(45)¯¯(1¡½)¡(1¡½¡2½2)¯1½1½2¯221¯¯¯½11½1¯¯¯¯½2½11¯Noticethat,foranAR(1)modelwithparametera,thenumera-toroftheexpressionfor®22iszero;andfor®33,thenumerator¡¢simpli¯estoa3+a5¡2a3¡a3a2¡1=0.ThesamplepartialACF,orSPACF,isjustthe¯nitesamplecounterpartofthetheoreticalPACF.Itcanbecomputedvia(43)butusingthesamplevalues^½i.Acomputationallymoree±cientmethodofcomputingthe®ssfromasetofcorrelationsisgivenbytheDurbin{Levinsonalgorithm;see,forexample,BrockwellandDavis(1991)andPollock(1999)foragooddiscussionandoriginalreferences.Thesmall{sampledis-tributionofthejointdensityoftheSPACFcanbeobtainedbytransformingthedensityoftheSACF;seeButlerandPaolella(1998)andthereferencesthereinfordetailsontherequiredJacobian. 1820.50.50.40.40.30.30.20.20.10.100−0.1−0.1−0.2−0.2−0.3−0.3−0.4−0.4−0.5−0.5024681012024681012Figure31:SACFandSPACFforanAR(1)process0.80.80.60.60.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.6−0.8−0.80510152005101520Figure32:ThetheoreticalACF(left)andPACFofthesubsetAR(5)modelwitha1=1:1,a3=¡0:6anda5=0:4Itcanbeshownthat,foriidnormaldata(andotheruncorrelatedprocesseswhichrelaxthenormalityassumption),T1=2®^isasymp-iitoticallystandardnormal.ExampleTheSACFandSPACFofarealizationofanAR(1)processwitha=0:5areshowninFigure31.ExampleConsidertheAR(5)processwithparametersa1=1:1,a2=0,a3=¡0:6,a4=0anda5=0:4.Inthiscase,someofthecoe±cientsarezero;thisisreferredtoasasubsetautoregressivemodel.TheACFandPACFareshowninFigure32.ObservethatitisnotthecasethatthesecondandfourthspikesinthePACFarezero,butonlythatitcutso®afterlag5. 183TheoreticalandSamplePACFforMA(1)Recallingthede¯nitionofpartialcorrelationandthecorrelationstructureofanMA(1)model,½¡½½½¡½½½¡½213122321121®22=½13j(2)=p22=p22=2(1¡½12)(1¡½23)(1¡½1)(1¡½1)1¡½1or³´2b¡1+b2¡b2®22=³´2=1+b2+b4:b1¡21+b¡¢From(44),®=b3=1+b2+b4+b6which,forb=0:5,is8=85.33Thepatternfor®kksuggeststhat(¡b)k(b2¡1)®kk=;1¡b2(k+1)whichisindeedtrue;see,forexample,BrockwellandDavis(1991,x3.4).Figure33plotsthetheoreticalACFandPACFcorrespond-ingtotheMA(1)modelwithb=0:5,sothat½1=0:4,showingthat,whiletheACFcutso®afterlag1,thePACFdiesoutexpo-nentially.0.50.50.40.40.30.30.20.20.10.100−0.1−0.1−0.2−0.2−0.3−0.3−0.4−0.4−0.5−0.5012345678012345678Figure33:ThetheoreticalACF(left)andPACF(right)fortheMA(1)modelwithb=0:5 184SACFandSPACFforAR,MAandARMAFora(stationary)AR(p)process,p¸1,theTACFdecaysexpo-nentially,10buttheTPACFcutso®"afterthepthspike,i.e.,itiszero.MattersarereversedforanMA(q)process:TheTACFcutso®aftertheqthspike,whiletheTPACFisnonzero,anddecaysexpo-nentially.Foramixed,i.e.,ARMAprocess,neithertheTACFortheTPACFcuto®.ThepracticalsideofthisresultisthatthesampleACFandPACFcanbeinspectedand,ifoneofthemappearstocuto®",thenpandqcanbeguessed.Ofcourse,inpractice,withrealdatawhichmaynotevenbefromastationarymodel,letaloneastationaryARMAmodel,letaloneapureARorpureMAmodel,mattersaren'talwayssoclearcut.Thedelicatesubjectofmodelidenti¯cationisdiscussedlater.10Itis,however,nottruethatallthespikesarenonzero.Asanexample,fortheAR(2)modelwitha1=1:2anda2=¡0:8,½2=0. 185ModelIdenti¯cation:Quotes!OnereasonwhyAkaikedoesnotaccepttheproblemofARMAorderselectionasthatofestimatinganunknowntrueorder,(m0,h0),say,isthatthereisnofundamentalreasonwhyatimeseriesneednecessarilyfollowa`true'ARMAmodel.(RajJ.Bhansali,1993)Thepurposeoftheanalysisofempiricaldataisnotto¯ndthetruemodel"|notatall.Instead,wewishto¯ndabestapproximatingmodel,basedonthedata,andthendevelopstatisticalinferencesfromthismodel.Dataanalysisinvolvesthequestion,Whatlevelofmodelcomplexitywillthedatasupport?"andbothunder{andover¯ttingaretobeavoided.(KennethP.BurnhamandDavidR.Anderson,2002,p.143) 186ModelIdenti¯cation:RealityDoingeconometricsisliketryingtolearnthelawsofelec-tricitybyplayingtheradio.(GuyOrcutt)Emphasizingthemessageintheabovetwoquotes,itisessentialtorealizethatanARMA(p;q)modelisnothingbutanapprox-imationtotheactual,unknowndatageneratingprocess.Therearenotrue"valuesofpandqwhichneedtobedetermined.Instead,valuesofpandqareselected(andthecorrespondingpa-rametersarethenestimated)whichprovideanacceptableapproxi-mationtothetrue,butunknown(andalmostalwaysunknowable)datageneratingprocess.TheARMAclassofmodelsisquiterichinthesensethat,evenwithp+qrelativelysmall,averywidevarietyofcorrelationstructuresarepossible.Thegoalofidenti¯cationistoselectthemostappropriatechoice(orchoices)ofpandq. 187WhatnottodoGiventhe°exibilityoftheautocorrelationstructurepossiblewithARMAmodels,itmightseemtemptingtojustpicklargeenoughvaluesofpandq,perhapsasafunctionoftheavailablesamplesize,soastoensurethattheautocorrelationstructureofthegivendatasetisarbitrarilycloselyreplicatedbytheARMAmodel.Weknowthatthisispossiblejustby¯ttinganMA(q)modelwithlargeenoughq,evenifthedataarenotgeneratedbyanMAmodel.AhighorderARmodelwillalsowork,andiseasiertoestimate.Theproblemwithsuchastrategyisthattheparametersneedtobeestimated,andthemorethereare,thelowerwillbetheiraccuracy.Furthermore,whensuchamodelisusedtomakeforecasts,ittendstoperforminadequately,ifnotdisastrously.Suchamodelissaidtobeover¯tted.Abettermodelwillem-bodytheprincipleofparsimony:Simplyput,LessisMore.Thegoalisto¯ndthesmallestvaluesofpandqwhichcaptureanadequateamount"ortheprimaryfeaturesof",thecorrelationstructure.Youarecorrectinhavingthefeelingthatthereisaconsiderableamountofsubjectivityinvolvedinthisactivity!Fortunately,someofthissubjectivityisremovable. 188CorrelogramAnalysisTherearetwothingsyouarebettero®notwatchinginthemaking:sausagesandeconometricestimates.(EdwardLeamer)WebeginwiththemethodwhichwaspopularizedbyBoxandJenk-ins,andinvolvesexaminingthesampleACF(SACF)andsamplePACF(SPACF)togetcandidatevaluesforpandq.Asdiscussedbefore,iftheSACFappearstocuto®",thenonecanpostulatethatthemodelisanMA(q),whereqistakentobethenumberofspikesbeforecuto®".Similarly,iftheSPACFcutso®,thenanAR(p)modelwouldbedeclared.Remember:Theasymptotic95%errorbandsshownontheSACFandSPACFareone-at-a-time95%con¯denceintervals,sothatoneexpectsonespikein20,onaverage,tofalloutsidetheintervalwhenthenullhypothesisofnoautocorrelationistrue.Whatonetypicallydoesinpracticeisaddsomepersonal,sub-jective,aprioribeliefsintothedecisionofwhichspikestodeemsigni¯cant(basedpresumablyontheculminationofexperience"onbehalfofthemodeler).Thesebeliefstypicallyincludeconsideringlow{orderspikestobemoreimportant(fornon{seasonaldataofcourse).Furthercomplicatingmattersisthecorrelationofthespikesinthecorrelograms,sothatsigni¯cant"spikestendtoariseinclusters. 189DetailedExampleOfcourse,ifthetrueprocesscomesfromamixedARMAmodel,thenneithercorrelogramcutso®!Figure34providesanexamplewitharti¯cialdata,consistingof100pointsgeneratedfromanARMA(2,2)modelwithparametersa=1:1,a=¡0:4,b=¡0:5,b=0:7,c=0and¾2=1.The1212toptwopanelsshowthetheoreticalACF(TACF)andtheoreticalPACF(TPACF)correspondingtotheprocess.Thesecondrowshowsatimeplotoftheactualdata.Thisparticularrealizationoftheprocessisinteresting(andnotunlikely),inthatcertainsegmentsofthedataappeartobefromadi®erentprocesses.Ofcourse,havinggeneratedtheprocessourselves,weknowitisindeedstationaryandwhatappeartobeanomaliesinthedataarejustartifactsofchance.Anappliedpractitioner,sayabuddingeconometrician,confrontedwiththisdatamightbeinclinedto¯ndoutwhat(say,macroeconomic)eventoccurrednearobservation35whichreversedadownwardtrendtoanupwardone,amidstclearperiodicbehavior,onlytochangeagaintoadownwardtrendwithoutperiodicbehavior,and,¯nally,crashed"nearobservation90,butbouncedbackabruptlyinarallyingtrend.(Thereareobviouslymanystructuralbreaksinthemodel,andtheseneedtobeinvestigated!")Thisillustratesthebene¯tofparsimoniousmodeling:Ifweweretointroducedummyexogenousvariablestoaccountforthehandfulofoutliers"inthedata,and/orusemoresophisticatedstructurestocapturetheapparentchangesinthemodel,etc.,itwouldallbefornought:Themodelarrivedatafterhoursordaysofseriousacademiccontemplation,research,andworkwouldbeutterlywrong,andwhileableto¯ttheobserveddatawell,wouldproduceunreliableforecasts,nottomentionafalseunderstandingofcausaleconomicrelationships. 1900.80.80.60.6TACFTPACF0.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.6−0.8−0.805101520051015206420−2−4−60204060801000.60.6SACFSPACF0.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.605101520051015200.80.80.60.6ETACFETPACF0.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.6−0.8−0.80510152005101520Figure34:ToppanelsarethetheoreticalcorrelogramscorrespondingtoastationaryandinvertibleARMA(2,2)modelwithparametersa1=1:1,a2=¡0:4,b1=¡0:5,b2=0:7,c=0and¾2=1.Thesecondrowshowsarealizationoftheprocess,withitssamplecorrelogramsplottedinthethirdrow.ThelastrowarethetheoreticalACFandPACFbutbasedontheestimatedARMA(2,2)modelofthedata. 191Thereadershouldnotgettheimpressionthatmost,ifnotalldatasetsareactuallystationary;onthecontrary,mostrealdatasetsaremostlikelynotstationary!Butthenatureofthenonstation-aritiesissodi±culttoguessat,thatsimple,parsimoniousmodelsshouldbepreferred.Indeed,actualstudiesofforecastqualitybasedonrealdataconsistentlycon¯rmthis(MakridakisandHibon,2000).SeealsoKeuzenkampandMcAleer(1997)andZellner(2001)fordiscussionsonthefutilityofcomplexmodelsforforecasting.Thethirdrowofplotsshowthesamplecorrelograms,whichdoindeedsomewhatresemblethetheoreticalones.BasedonthedecayofthesampleACFandthecuto®ofthePACFatlag3,itwouldseemthatanAR(3)modelwouldbeappropriate.ThelastrowshowsthetheoreticalcorrelogramswhichcorrespondtotheestimatedARMA(2,2)model(assumingaknownmeanofzero).TheMLEvalues(andapproximatestandarderrorsinparen-theses)are^a1=0:946(0:12),^a2=¡0:245(0:12),^b1=¡0:364(0:076),^b2=0:817(0:078)and^¾=0:966(0:069).Noticethatthesecorrelogramsareclosertothetrueonesthanarethesamplecorrelograms.Thisisquitereasonable,becausemoreinformationisusedintheirconstruction(inparticular,knowledgeoftheparametricmodelbeinganARMA(2,2)andmaximumlike-lihoodestimation,asopposedsimplytosamplemoments).Ofcourse,thisknowledgeofpandqisnotrealisticinpracticalset-tings. 1920.60.6SACF(1−50)SPACF(1−50)0.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.605101520051015200.80.80.60.6SACF(51−100)SPACF(51−100)0.40.40.20.200−0.2−0.2−0.4−0.4−0.6−0.6−0.8−0.80510152005101520Figure35:Aninformalgraphicaltestforcovariancestationarityistocomputethesamplecorrelogramsfornonoverlappingsegmentsofthedata.Inpractice,itisalsoagoodideatocomputethecorrelogramsfordi®erentsegmentsofthedata,thenumberofsegmentsdepend-ingontheavailablesamplesize.Ifthedataarefromastation-aryprocess,thentheSACFsforthedi®erentsegmentsshouldberoughly"thesame(andsimilarlyfortheSPACFs).The¯gureshowsthesamplecorrelogramscorrespondingtothetwohalvesofthedataunderinvestigation.Whiletheyclearlyhavecertainsimilarities,noticethattheSACFfromthe¯rsthalfappearstocuto®aftertwolargespikes(sug-gestinganMA(2)model),whiletheSACFforthesecondhalfdiesoutgradually,indicativeofARorARMAprocesswithp>0.As-sumingstationarity,weaddtoourcollectionoftentativemodelsanMA(2)andanARMA(1,1). 193TestingThethreegoldenrulesofeconometricsaretest,test,andtest.(DavidHendry)...econometrictesting,asagainstestimation,isnotworthanythingatall.Itsmarginalproductiszero.Itisaslackvariable.(Deirde[formerlyDonald]McCloskey)Keepinmindthethreemostimportantaspectsofrealdataanalysis:compromise,compromise,andcompromise.(EdwardLeamer)Inthiscontext,itseemsrathernaturaltoconductahypothesistestonaparameterinquestion,wherethenullhypothesisisthatitiszeroandthealternativeisthatitisnonzero.Thisisverystraightforwardwhenassumingthevalidityoftheasymp-toticdistributioninsmallsamples.Inparticular,letT^i=µ^i=SE(cµ^i),i=1;:::;p+q,betheithstandardizedparameterestimate.(TheuseofTisreminiscentofthet{statisticfromlinearregression.)Thenthesize{®hypothesistestH0:µi=0versusH1:µi6=0wouldrejectthenullifthep{valueassociatedwithjT^ijissmallenough.Equivalently,onechecksifzeroiscontainedinthe100(1¡®)%con¯denceintervalofµi.Forthedatasetwearediscussing,the95%con¯denceintervalsbasedontheasymptoticnormalityoftheestimatesare(0:7050.ThismodelformulationquiteconvenientlynestsatleastsevenGARCH(r;s)modelspreviouslyproposedintheliterature.Forexample,with±=1;°=0,(53)reducesto(52),and°6=0al-lows¾ttorespondasymmetricallytopositiveandnegativeshocks. 209Model(53)givesrisetoastrictlystationaryseriesforparametervaluessuchthatXrXsV:=·ici+dj·1(54)i=1j=1where±·i:=E(jzj¡°iz)(55)dependsonthedensityspeci¯cationfZ(¢).AnintegratedA-PARCH,orIA-PARCHmodelisgivenbyV=1while,forV>1,themodelwillbeexplosive.Whatremainstobespeci¯edistheinnovationsdensityfZ(¢).Fornormalinnovationsand°i=°,straightforwardcalculationyieldshiµ¶1±±±¡1±+1·=p(1+°)+(1¡°)22¡;2¼2where¡(¢)denotesthegammafunction. 210ModelGoodnessofFitTohelpdistinguishbetweencompeting(usuallynon-nested)con-ditionalmodels,avarietyofso-calledgoodness-of-¯tstatisticscanbeused.TheseincludethemeasurescommoninARMAanalysis(AIC,BIC,etc.),withthethecorrectedAICbeingquitepopular:2T(k+1)AICC=¡2L+;(56)T¡k¡2wherekdenotesthenumberofestimatedparametersandTthenumberofobservations.Thesemeasuresarebasedonthelikeli-hoodandpenalizeforlargenumberofparameters.Measuresbasedontheempiricalcumulativedensityfunction(ecdf)includeKolmogorov-SmirnovdistanceKD=100£supjFs(x)¡F^(x)j;(57)x2RwhereF^(x)denotesthecdfoftheestimatedparametricdensity,andFs(x)istheempiricalsampledistribution,i.e.,Tµ¶¡1Xrt¡¹^tFs(x)=TI(¡1;x]¾^tt=1whereI(¢)istheindicatorfunction.Thisstatisticisarobustmeasureinthesensethatitfocusesonlyonthemaximumdeviationbetweenthesampleand¯tteddistributions.TherelatedAnderson{Darlingstatistic:jFs(x)¡F^(x)jAD=supr³´(58)x2RF^(x)1¡F^(x)isbasedonrelativediscrepanciesandthereforemoreusefulifin-terestcentersprimarilyonthelefttailoftheconditionalreturndistribution(asin¯nancialapplications). 211ForecastingAsopposedtotheconventionalmean-variancecriterionparadigmformaking¯nancialdecisions,newerriskmanagementconceptsfor¯nancialinstitutionsareconcernedwiththevalue-at-riskofagiven¯nancialposition.Wefocusattentionhereonthethresholdzforwhicha¯nancialpo-sitionhperiodsinthefuturewillnotfallwithprobability(1¡°).Thatis,thevalue-at-riskimpliedbyaparticularmodel,M,forfutureperiodt+hisgivenbyzMsuchthatt+h¡¢MPrrt+h·¡zt+h(°)=°;where°isspeci¯ed(bytheinvestor)andz=zM(°)ispredictedt+hfromthemodel.Ifreturnswereiidnormal,onlythemeanandvarianceofthere-turnsneedbecomputedinordertospecifyz.Forconditionallyheteroskedasticmodelswithnonnormalinnova-tiondistributions,thecomputationofzwillrequirespeci¯cationoftheconditionaldensityµ¯¶f^r¡¹^t+hjt¯¯µ^t+hjt(r)=f¯t;rt;rt¡1;:::;¾^t+hjtwhereµ^treferstotheestimatedparametervectorusingthesampleinformationuptoandincludingperiodt;and^¹t+hjtand^¾t+hjtdenotetherespectiveforecastsof¹and¾attimet+hgiventhe¯rsttobservations. 212Becausetheheteroscedasticitydoesnotin°uencethemeanofthetimeseries,forecast^¹t+`jt,`=1;:::;h,isjust^rt+`computedrecursivelyfromthestandardARMArecursive`-stepaheadfore-castof(51)substitutingE(²j)=0forj>t;while,usingtheA-PARCH(r;s)model(53),^¾t+`jtisrecursivelyevaluatedfromXrXs¾^±=^c+c^Ei+d^¾^±;t+`jt0it+`¡ijt+`¡jjti=1j=1`=1;:::;h,where^¾kjt,k=1;:::;t,arethe¯lteredin-samplevolatilities;^¾kjt,k>t,denoterecursivelycomputedoutofsampleforecasts;(±^i(j²kj¡°^i²k);ifk·t;Ek=±^±^E(j²kj¡°^i²k)=^¾k·i;ifk>t;and²k,k·t,denotethe¯lteredinnovations.Value·iisgivenby(55)evaluatedwiththeappropriateestimatedparameters.Predic-tionsusingthestandardGARCHmodel(52)areclearlyaspecialcase. 213Inpractice,anh-lengthhold-outsamplecanbetaken,i.e.,ob-servationst=T¡h+1;:::;T,andthequalityoftheforecastcanbejudged.Foreacht,themodelparametersareupdated(re-estimatedviaMLestimation)because,inactualapplications,onewouldusealltheavailabledatatomakeinferenceaboutthefuture.Foracorrectlyspeci¯edmodelweexpect100°%oftheobservedrt+1-valuestobelessthanorequaltotheimpliedthreshold-values¡zM(°).t+1Iftheobservedfrequency1XT¡1M°^:=1000I(¡1;¡ztM+1(°)](rt+1)(59)t=T¡1000isless(higher)than°,thenmodelMtendstooverestimate(un-derestimate)theriskofthecurrencyposition;i.e.,theimpliedabsolutez-valuestendtobetoolarge(small).