2 markovchain package discrete Markov chains in R, perform efficient matrices powers igraph Csardi and Nepusz 2006 to perform pretty plotting. of markovchain objects and matlab Roebuck 2011 that contains functions for matrix ma. nagement and calculations that emulate those within MATLAB environment Moreover other. scientific softwares provide functions specifically designed to analyze DTMC as Mathematica. 9 Wolfram Research 2013b, The paper is structured as follows Section 2 briefly reviews mathematics and definitions re. garding DTMC Section 3 discusses how to handle and manage Markov chain objects within. the package Section 4 and Section 5 show how to perform probabilistic and statistical model. ling while Section 6 presents some applied examples from various fields analyzed by means. of the markovchain package,2 Review of core mathematical concepts. 2 1 General Definitions, A DTMC is a sequence of random variables X1 X2 Xn characterized by the Markov. property also known as memoryless property see Equation 1 The Markov property states. that the distribution of the forthcoming state Xn 1 depends only on the current state Xn. and doesn t depend on the previous ones Xn 1 Xn 2 X1. P r Xn 1 xn 1 X1 x1 X2 x2 Xn xn P r Xn 1 xn 1 Xn xn 1. The set of possible states S s1 s2 sr of Xn can be finite or countable and it is named. the state space of the chain, The chain moves from one state to another this change is named either transition or step. and the probability pij to move from state si to state sj in one step is named transition. probability,pij P r X1 sj X0 si 2, The probability of moving from state i to j in n steps is denoted by pij P r Xn sj X0 si. A DTMC is called time homogeneous if the property shown in Equation 3 holds Time. homogeneity implies no change in the underlying transition probabilities as time goes on. P r Xn 1 sj Xn si P r Xn sj Xn 1 si 3, If the Markov chain is time homogeneous then pij P r Xk 1 sj Xk si and. pij P r Xn k sj Xk si where k 0, The probability distribution of transitions from one state to another can be represented into. a transition matrix P pij i j where each element of position i j represents the transition. probability pij E g if r 3 the transition matrix P is shown in Equation 4. p11 p12 p13,P p21 p22 p23 4,p31 p32 p33, G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 3. The distribution over the states, P can be written in the form of a stochastic row vector x the. term stochastic means that i xi 1 xi 0 e g if the current state of x is s2 x 0 1 0. As a consequence the relation between x 1 and x 0 is x 1 x 0 P and recursively we get. x 2 x 0 P 2 and x n x 0 P n n 0, DTMC are explained in most theory books on stochastic processes see Bre maud 1999 and. Dobrow 2016 for example Valuable references online available are Konstantopoulos 2009. Snell 1999 and Bard 2000,2 2 Properties and classification of states. A state sj is said accessible from state si written si sj if a system starting in state si has. a positive probability to reach the state sj at a certain point i e n 0 pnij 0 If both. si sj and sj si then si and sj are said to communicate. A communicating class is defined to be a set of states that communicate A DTMC can be. composed by one or more communicating classes If the DTMC is composed by only one. communicating class i e if all states in the chain communicate then it is said irreducible. A communicating class is said to be closed if no states outside of the class can be reached. from any state inside it, If pii 1 si is defined as absorbing state an absorbing state corresponds to a closed com. municating class composed by one state only, The canonical form of a DTMC transition matrix is a matrix having a block form where the. closed communicating classes are shown at the beginning of the diagonal matrix. A state si has period ki if any return to state si must occur in multiplies of ki steps that is. ki gcd n P r Xn si X0 si 0 where gcd is the greatest common divisor If ki 1. the state si is said to be aperiodic else if ki 1 the state si is periodic with period ki Loosely. speaking si is periodic if it can only return to itself after a fixed number of transitions ki 1. or multiple of ki else it is aperiodic, If states si and sj belong to the same communicating class then they have the same period ki. As a consequence each of the states of an irreducible DTMC share the same periodicity This. periodicity is also considered the DTMC periodicity It is possible to classify states according. to their periodicity Let T x x is the number of periods to go back to state x knowing that. the chain starts in x, A state x is recurrent if P T x x 1 equivalently P T x x 0 In. 1 A state x is null recurrent if in addition E T x x. 2 A state x is positive recurrent if in addition E T x x. 3 A state x is absorbing if in addition P T x x 1 1. A state x is transient if P T x x 1 equivalently P T x x 0. It is possible to analyze the timing to reach a certain state The first passage time or hitting. time from state si to state sj is the number Tij of steps taken by the chain until it arrives. for the first time to state sj given that X0 si The probability distribution of Tij is defined. 4 markovchain package discrete Markov chains in R,by Equation 5. hij n P r Tij n P r Xn sj Xn 1 6 sj X1 6 sj X0 si 5. and can be found recursively using Equation 6 given that hij n pij. hij n pik hkj n 1 6, A commonly used quantity related to h is its average value i e the mean first passage time. also expected hitting time namely h ij n 1 n hij, If in the definition of the first passage time we let si sj we obtain the first recurrence time. Ti inf n 1 Xn si X0 si We could also ask ourselves which is the mean recurrence. time an average of the mean first recurrence times. ri k P Ti k, Revisiting the definition of recurrence and transience a state si is said to be recurrent if it is. visited infinitely often i e P r Ti X0 si 1 On the opposite si is called transient. if there is a positive probability that the chain will never return to si i e P r Ti X0. Given a time homogeneous Markov chain with transition matrix P a stationary. P distribution, z is a stochastic row vector such that z z P where 0 zj 1 j and j zj 1. If a DTMC Xn is irreducible and aperiodic then it has a limit distribution and this distri. bution is stationary As a consequence if P is the k P k transition matrix of the chain and. z z1 zk is the unique eigenvector of P such that ki 1 zi 1 then we get. lim P n Z 7, where Z is the matrix having all rows equal to z The stationary distribution of Xn is. represented by z, A matrix A is called primitive if all of its entries are strictly positive and we write it A 0 If. the transition matrix P for a DTMC has some primitive power i e it exists m 0 P m 0. then the DTMC is said to be regular In fact being regular is equivalent to being irreducible. and aperiodic All regular DTMCs are irreducible The counterpart is not true. Given two absorbing states sA source and sB sink the committor probability qj is the. probability that a process starting in state si is absorbed in state sB rather than sA Noe. Schu tte Vanden Eijnden Reich and Weikl 2009 It can be computed via. AB AB AB AB,qj Pjk qk with qA 0 and qB 1 8, G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 5. Note we can also define the hitting probability from i to j as the probability of ever reaching. the state j if our initial state is i,hi j P r Tij hij 9. In a DTMC with finite set of states we know that a transient state communicates at least. with one recurrent state If the chain starts in a transient element once it hits a recurrent. state it is going to be caught in its recurrent state and we cannot expect it would go back. to the initial state Given a transient state i we can define the absorption probability to the. recurrent state j as the probability that the first recurrent state that the Markov chain visits. and therefore gets absorbed by its recurrent class is j fi j We can also define the mean. absorption time as the mean number of steps the transient state i would take until it hits any. recurrent state bi,2 3 A short example, Consider the following numerical example Suppose we have a DTMC with a set of 3 possible. states S s1 s2 s3 Let the transition matrix be,0 5 0 2 0 3. P 0 15 0 45 0 4 10,0 25 0 35 0 4, In P p11 0 5 is the probability that X1 s1 given that we observed X0 s1 is 0 5 and so. on It is easy to see that the chain is irreducible since all the states communicate it is made. by one communicating class only, Suppose that the current state of the chain is X0 s2 i e x 0 0 1 0 then the probability. distribution of states after 1 and 2 steps can be computed as shown in Equations 11 and. 0 5 0 2 0 3,x 1 0 1 0 0 15 0 45 0 4 0 15 0 45 0 4 11. 0 25 0 35 0 4,0 5 0 2 0 3, x n x n 1 P 0 15 0 45 0 4 0 15 0 45 0 4 0 2425 0 3725 0 385 12. 0 25 0 35 0 4, If we were interested in the probability of being in the state s3 in the second step then. P r X2 s3 X0 s2 0 385,6 markovchain package discrete Markov chains in R. 3 The structure of the package,3 1 Creating markovchain objects. The package is loaded within the R command line as follows. R library markovchain,Loading required package matlab. Attaching package matlab,The following object is masked from package stats. The following objects are masked from package utils. The following object is masked from package base, The markovchain and markovchainList S4 classes Chambers 2008 are defined within the. markovchain package as displayed,Class markovchain package markovchain. Name states byrow transitionMatrix name,Class character logical matrix character. Class markovchainList package markovchain,Name markovchains name. Class list character, The first class has been designed to handle homogeneous Markov chain processes while. the latter which is itself a list of markovchain objects has been designed to handle non. homogeneous Markov chains processes, Any element of markovchain class is comprised by following slots. G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 7. 1 states a character vector listing the states for which transition probabilities are. 2 byrow a logical element indicating whether transition probabilities are shown by row. or by column, 3 transitionMatrix the probabilities of the transition matrix. 4 name optional character element to name the DTMC. The markovchainList objects are defined by following slots. 1 markovchains a list of markovchain objects, 2 name optional character element to name the DTMC. The markovchain objects can be created either in a long way as the following code shows. R weatherStates c sunny cloudy rain,R byRow TRUE,R weatherMatrix matrix data c 0 70 0 2 0 1. R 0 3 0 4 0 3,R 0 2 0 45 0 35 byrow byRow nrow 3,R dimnames list weatherStates weatherStates. R mcWeather new markovchain states weatherStates byrow byRow. R transitionMatrix weatherMatrix name Weather,or in a shorter way displayed below. R mcWeather new markovchain states c sunny cloudy rain. R transitionMatrix matrix data c 0 70 0 2 0 1,R 0 3 0 4 0 3. R 0 2 0 45 0 35 byrow byRow nrow 3,R name Weather, When new markovchain is called alone a default Markov chain is created. R defaultMc new markovchain, The quicker way to create markovchain objects is made possible thanks to the implemented. initialize S4 method that checks that, the transitionMatrix to be a transition matrix i e all entries to be probabilities and. either all rows or all columns to sum up to one, the columns and rows names of transitionMatrix to be defined and to coincide with. states vector slot, The markovchain objects can be collected in a list within markovchainList S4 objects as. following example shows,8 markovchain package discrete Markov chains in R. R mcList new markovchainList markovchains list mcWeather defaultMc. R name A list of Markov chains,3 2 Handling markovchain objects. Table 1 lists which methods handle and manipulate markovchain objects. Method Purpose,Direct multiplication for transition matrices. Compute the power markovchain of a given one, Direct access to the elements of the transition matrix. Equality operator between two transition matrices, Inequality operator between two transition matrices. as Operator to convert markovchain objects into data frame and. table object,dim Dimension of the transition matrix. names Equal to states,names Change the states name. name Get the name of markovchain object,name Change the name of markovchain object. plot plot method for markovchain objects,print print method for markovchain objects. show show method for markovchain objects, sort sort method for markovchain objects in terms of their states. states Name of the transition states, t Transposition operator which switches byrow slot value and modifies. the transition matrix coherently, Table 1 markovchain methods for handling markovchain objects. The examples that follow shows how operations on markovchain objects can be easily per. formed For example using the previously defined matrix we can find what is the probability. distribution of expected weather states in two and seven days given the actual state to be. R initialState c 0 1 0,R after2Days initialState mcWeather mcWeather. R after7Days initialState mcWeather 7,R after2Days. sunny cloudy rain,1 0 39 0 355 0 255,R round after7Days 3. sunny cloudy rain,1 0 462 0 319 0 219, G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 9. A similar answer could have been obtained defining the vector of probabilities as a column. vector A column defined probability matrix could be set up either creating a new matrix. or transposing an existing markovchain object thanks to the t method. R initialState c 0 1 0,R after2Days t mcWeather t mcWeather initialState. R after7Days t mcWeather 7 initialState,R after2Days. sunny 0 390,cloudy 0 355,rain 0 255,R round after7Days 3. sunny 0 462,cloudy 0 319,rain 0 219, The initial state vector previously shown can not necessarily be a probability vector as the. code that follows shows,R fvals function mchain initialstate n. R out data frame,R names initialstate names mchain. R for i in 0 n,R iteration initialstate mchain i,R out rbind out iteration. R out cbind out i seq 0 n,R out out c 4 1 3,R return out. R fvals mchain mcWeather initialstate c 90 5 5 n 4. i sunny cloudy rain,1 0 90 00000 5 00000 5 00000,2 1 65 50000 22 25000 12 25000. 3 2 54 97500 27 51250 17 51250,4 3 50 23875 29 88063 19 88062. 5 4 48 10744 30 94628 20 94628, Basic methods have been defined for markovchain objects to quickly get states and transition. matrix dimension, 10 markovchain package discrete Markov chains in R. R states mcWeather,1 sunny cloudy rain,R names mcWeather. 1 sunny cloudy rain,R dim mcWeather, Methods are available to set and get the name of markovchain object. R name mcWeather,R name mcWeather New Name,R name mcWeather. 1 New Name, Also it is possible to alphabetically sort the transition matrix. R markovchain sort mcWeather, A 3 dimensional discrete Markov Chain defined by the following states. cloudy rain sunny, The transition matrix by rows is defined as follows. cloudy rain sunny,cloudy 0 40 0 30 0 3,rain 0 45 0 35 0 2. sunny 0 20 0 10 0 7, A direct access to transition probabilities is provided both by transitionProbability met. hod and method,R transitionProbability mcWeather cloudy rain. R mcWeather 2 3, G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 11. The transition matrix of a markovchain object can be displayed using print or show methods. the latter being less verbose Similarly the underlying transition probability diagram can. be plotted by the use of plot method as shown in Figure 1 which is based on igraph. package Csardi and Nepusz 2006 plot method for markovchain objects is a wrapper of. plot igraph for igraph S4 objects defined within the igraph package Additional parameters. can be passed to plot function to control the network graph layout There are also diagram. and DiagrammeR ways available for plotting as shown in Figure 2 The plot function also uses. communicatingClasses function to separate out states of different communicating classes. All states that belong to one class have same color. R print mcWeather,sunny cloudy rain,sunny 0 7 0 20 0 10. cloudy 0 3 0 40 0 30,rain 0 2 0 45 0 35,R show mcWeather. A 3 dimensional discrete Markov Chain defined by the following states. sunny cloudy rain, The transition matrix by rows is defined as follows. sunny cloudy rain,sunny 0 7 0 20 0 10,cloudy 0 3 0 40 0 30. rain 0 2 0 45 0 35,Attaching package igraph, The following objects are masked from package stats. decompose spectrum,The following object is masked from package base. Warning package diagram was built under R version 4 0 0. Loading required package shape, Import and export from some specific classes is possible as shown in Figure 3 and in the. following code, 12 markovchain package discrete Markov chains in R. cloudy 0 4,Figure 1 Weather example Markov chain plot. 0 4 0 2 0 7,cloudy sunny, Figure 2 Weather example Markov chain plot with diagram. G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 13. R mcDf as mcWeather data frame,R mcNew as mcDf markovchain. t0 t1 prob,1 sunny sunny 0 70,2 sunny cloudy 0 20,3 sunny rain 0 10. 4 cloudy sunny 0 30,5 cloudy cloudy 0 40,6 cloudy rain 0 30. 7 rain sunny 0 20,8 rain cloudy 0 45,9 rain rain 0 35. R mcIgraph as mcWeather igraph,R if requireNamespace msm quietly TRUE. R require msm,R Q rbind c 0 0 25 0 0 25,R c 0 166 0 0 166 0 166. R c 0 0 25 0 0 25,R c 0 0 0 0, R cavmsm msm state years subject PTNUM data cav qmatrix Q death 4. R msmMc as cavmsm markovchain,R message msm unavailable. Loading required package msm, Warning package msm was built under R version 4 0 0. Unnamed Markov chain, A 4 dimensional discrete Markov Chain defined by the following states. State 1 State 2 State 3 State 4, The transition matrix by rows is defined as follows. State 1 State 2 State 3 State 4, State 1 0 853958721 0 08836953 0 01475543 0 04291632. State 2 0 155576908 0 56663284 0 20599563 0 07179462. State 3 0 009903994 0 07853691 0 65965727 0 25190183. State 4 0 000000000 0 00000000 0 00000000 1 00000000. R if requireNamespace etm quietly TRUE,R library etm. 14 markovchain package discrete Markov chains in R. R data sir cont, R sir cont sir cont order sir cont id sir cont time. R for i in 2 nrow sir cont,R if sir cont id i sir cont id i 1. R if sir cont time i sir cont time i 1,R sir cont time i 1 sir cont time i 1 0 5. R tra matrix ncol 3 nrow 3 FALSE,R tra 1 2 3 TRUE,R tra 2 c 1 3 TRUE. R tr prob etm sir cont c 0 1 2 tra cens 1,R etm2mc as tr prob markovchain. R message etm unavailable,Unnamed Markov chain, A 3 dimensional discrete Markov Chain defined by the following states. The transition matrix by rows is defined as follows. 0 0 0000000 0 5000000 0 5000000,1 0 5000000 0 0000000 0 5000000. 2 0 3333333 0 3333333 0 3333333, Coerce from matrix method as the code below shows represents another approach to create. a markovchain method starting from a given squared probability matrix. R myMatr matrix c 1 8 1 2 6 2 3 4 3 byrow TRUE ncol 3. R myMc as myMatr markovchain,Unnamed Markov chain, A 3 dimensional discrete Markov Chain defined by the following states. The transition matrix by rows is defined as follows. s1 0 1 0 8 0 1,s2 0 2 0 6 0 2,s3 0 3 0 4 0 3, Non homogeneous Markov chains can be created with the aid of markovchainList object. The example that follows arises from health insurance where the costs associated to patients. G A Spedicato T S Kang S B Yalamanchi D Yadav I Cordo n 15. Import Export from and to markovchain objects,markovchain igraph. sparseMatrix, Figure 3 The markovchain methods for import and export. in a Continuous Care Health Community CCHC are modeled by a non homogeneous Mar. kov Chain since the transition probabilities change by year Methods explicitly written for. markovchainList objects are print show dim and,R stateNames c H I D. R Q0 new markovchain states stateNames, R transitionMatrix matrix c 0 7 0 2 0 1 0 1 0 6 0 3 0 0 1. R byrow TRUE nrow 3 name state t0,R Q1 new markovchain states stateNames. R transitionMatrix matrix c 0 5 0 3 0 2 0 0 4 0 6 0 0 1. R byrow TRUE nrow 3 name state t1,R Q2 new markovchain states stateNames. R transitionMatrix matrix c 0 3 0 2 0 5 0 0 2 0 8 0 0 1. R byrow TRUE nrow 3 name state t2,R Q3 new markovchain states stateNames. R transitionMatrix matrix c 0 0 1 0 0 1 0 0 1,R byrow TRUE nrow 3 name state t3. R mcCCRC new markovchainList markovchains list Q0 Q1 Q2 Q3. R name Continuous Care Health Community,R print mcCCRC. Continuous Care Health Community list of Markov chain s. Markovchain 1, A 3 dimensional discrete Markov Chain defined by the following states. The transition matrix by rows is defined as follows.

4 The PayBase ACH module adds electronic payment and collection functionality to the PayBase platform, turning a typically labor intensive, costly and multi-step

Table of Contents vii 4.4.4 Effects on fine root growth, mycorrhiza and vitality..... 94 4.5 Effects of wood ash application on ground 4.5.1 Species composition ...

This book provides a self-contained introduction to ordinary di erential equations and dynamical systems suitable for beginning graduate students. In the rst part we begin with some simple examples of explicitly solvable equations and a rst glance at qualitative methods. Then we prove the fundamental results concerning the initial value problem: existence, uniqueness, extensibility, dependence ...

part by "dressing up" as an avatar. There are hundreds of avatars to choose from, in- cluding penguins and real people. Once avatars have entered a world, they can ex- plore it and chat with other avatars. Color Plate 6 Figure 5.3 Examples of aesthetically pleasing interactive products: iMac, Nokia cell phone and IDEO's digital radio for the BBC. 1 Figure 5.9 Virtual screen characters: (a ...

looking to expand still further. We hope you find our insights useful and that it will help you to succeed in the world's fastest growing market. The China Factor: Doing Business in China 1. Foreign Investment in the PRC Since China's WTO entrance, it has taken major steps to liberalize its trade policy, expanding its "open-door" policy to make its investment environment both more welcoming ...

Places Bylaw 2019, and proposed revocation of the Nuisances Bylaw 2008" as considered by the Policy Committee on 27 July 2019. The effective date of the bylaws (resolution no. 6) has been changed from 23 August 2019 to 30 September 2019, to reflect the new Council meeting date of 29 August and to give

In addition, a Controlled Copy is available in the Master Control Room procedure binders at the ISO. Hard Copy Is Uncontrolled Revision 30, Effective Date: May 8, 2020 Page 1 of 57 ISO-NE PUBLIC ISO New England Operating Procedure No. 14 - Technical Requirements for Generators, Demand Response Resources, Asset Related Demands and Alternative Technology Regulation Resources Effective Date: May ...

SOP training Evaluations Responsibilities Document all training. Quality Assurance Teams A team should be formed to identify, evaluate, and control feed and food safety hazards. PURCHASING AND RECEIVING. Quality Assurance Manual Purchasing and Receiving Overview/Objective Producing safe feed of desired quality starts with the grains and ingredients provided by suppliers. Purchasing and ...