Scholarly article on topic 'Multi-agent Systems Modelling for Evaluating Joint Delivery Systems'

Multi-agent Systems Modelling for Evaluating Joint Delivery Systems Academic research paper on "Economics and business"

CC BY-NC-ND
0
0
Share paper
OECD Field of science
Keywords
{"Multi-agent systems" / "joint delivery systems" / "city logistics" / "reinforcement learning"}

Abstract of research paper on Economics and business, author of scientific article — Ornkamon Wangapisit, Eiichi Taniguchi, Joel S.E. Teo, Ali Gul Qureshi

Abstract This paper presents the Multi-Agent System (MAS) model for evaluating city logistics measures like the joint delivery systems and car parking management. This research is directed at evaluating possible measures that will provide logistics efficiency in a city faced with congested urban traffic conditions. The focus of the study is on the interaction and cooperation between urban freight stakeholders when city logistics measures are implemented and to investigate the behaviour of stakeholders with their objectives. The preliminary results of the model show that the joint delivery system and car parking management have the potential for improving environmental issues. In addition, the subsidies by the shopping street association and car parking management policy of the government can increase the frequency of UDC usage.

Academic research paper on topic "Multi-agent Systems Modelling for Evaluating Joint Delivery Systems"

Available online at www.sciencedirect.com

ScienceDirect

Procedía - Social and Behavioral Sciences 125 (2014) 472 - 483

8th International Conference on City Logistics

Multi-Agent Systems Modelling For Evaluating Joint Delivery

Systems

Ornkamon Wangapisita*, Eiichi Taniguchia, Joel S.E. Teoa, Ali Gul Qureshia

aDepartment of Urban Management, KyotoUniversity, Nishikyo-ku, Kyoto, 615-8540, Japan

Abstract

This paper presents the Multi-Agent System (MAS) model for evaluating city logistics measures like the joint delivery systems and car parking management. This research is directed at evaluating possible measures that will provide logistics efficiency in a city faced with congested urban traffic conditions. The focus of the study is on the interaction and cooperation between urban freight stakeholders when city logistics measures are implemented and to investigate the behaviour of stakeholders with their objectives. The preliminary results of the model show that the joint delivery system and car parking management have the potential for improving environmental issues. In addition, the subsidies by the shopping street association and car parking management policy of the government can increase the frequency of UDC usage.

©2014 The Authors. Published by Elsevier Ltd.

Selectionand peer-review under responsibility oftheOrganisingCommitteeofthe 8thInternationalConference on CityLogistics.

Keywords: Multi-agent systems; joint delivery systems; city logistics; reinforcement learning

1. Introduction

Previous research in city logistics has attempted to model and provide solutions to businesses and the community by ensuring the optimum efficiency of goods delivery, reliability and customer service, while reducing the negative environmental issues, such as, air pollution emissions, energy consumption and traffic congestion. One solution to improve and reduce the urban freight logistics problems is to introduce urban distribution centres

* Corresponding author. Tel.: +81-75-3833230; fax: +81-75-9503800. E-mail address: ornkamon.w@kiban.kuciv.kyoto-u.ac.jp

1877-0428 © 2014 The Authors. Published by Elsevier Ltd.

Selection and peer-review under responsibility of the Organising Committee of the 8th International Conference on City Logistics. doi:10.1016/j.sbspro.2014.01.1489

(UDCs) (Dablanc, 2007). The UDC is a promising concept, where the loads of delivery trucks from different carriers are consolidated at a single facility and transferred to new trucks to increase the load factor and to allow for easier time-windowed operations to avoid traffic congestion (Quak and Koster, 2009). The main factors contributing to a successful implementation of an UDC are its location in or near the city, subsidy collection, service cost of the UDC, shorter delay in delivery time and the collaborative relationship between the shippers and freight carriers. Furthermore, it is found that the most challenging factor is the collaboration between all involved stakeholders within the complex operation (van Duin, Kolck, Anand, Tavasszy and Taniguchi, 2012). On top of this, another associated city logistics problem is the illegally parked vehicles that are rampant in urban areas for the purpose of loading/ unloading activities. With limited parking spaces, many vehicles park illegally beside the road, which leads to traffic safety problems and deterioration of environmental issues.

Several researches have studied urban freight logistics by using the MAS modelling approach to evaluate city logistics measures, for instance, road pricing, toll pricing, truck ban, time windows restrictions, load factor control, operation subsidies and urban distribution centre usage (Antal, 2010; van Duin, Kolck, Anand, Tavasszy and Taniguchi, 2012; Vita and Janis, 2005; Teo, Taniguchi and Qureshi, 2012, Tamagawa, Taniguchi and Yamada, 2010; Taniguchi, Yamada and Okamoto, 2007). However, the interaction among the stakeholders of the joint delivery system and the consideration of illegally parked vehicles are intended to be studied in this research. The aim of this research is to explore if the implementation of joint delivery system and car parking management as city logistics measures can lead to an overall system benefit for all stakeholders by using the MAS model with reinforcement learning.

2. Objectives

The objective of this paper is to study the effect of city logistics measures consisting of the joint delivery systems, an urban distribution centre, and parking space restriction. To study the behaviour of urban freight stakeholders and their interaction, which is affected by the policy measures, the MAS modelling approach is used to represent their multi-objective environment. This paper discusses the MAS in the context of city logistics measures that are aimed at changing the stakeholders' behaviour. In addition, the performance measures evaluated will include truck emissions and other costs of the stakeholders.

3. Methodology

The MAS model for evaluating the joint delivery systems requires the identification of stakeholders, which include the logistics communities or logistics associations. Stakeholders are individuals, who belong to various identified "communities" and whose lives or businesses are affected by particular policies. Similarly, the policies may also affect the environment and transportation costs, which may ultimately affect the end consumers. The stakeholders identified in a joint delivery system include the freight carriers, a neutral carrier, shop owners, residents, administrators and logistics association that consolidate goods from various freight carriers and load it onto a neutral carrier, to dispatch to shop owners, by considering operation costs, truck assignment and time window. The urban freight logistics experts have emphasized the importance of the engagement of the stakeholders in terms of greater urban distribution centre usage, environmental issues as well as the potential of the joint delivery systems. The MAS model framework consists of the vehicle routing problem with soft time window VRPSTW (Qureshi, 2008) and the behavioural interaction among stakeholders with reinforcement learning model. The interaction among the stakeholders can be described using MAS interaction model as shown in Fig. 1. Fig. 2 shows a modified MAS model framework with vehicle routing and scheduling problems with time windows by Tamagawa, Taniguchi and Yamada (2010), which consists of two sub-models. One is the learning model for stakeholders, and the other is the model for vehicle routing and scheduling problems with soft time window (VRPSTW). The learning model evaluates the behaviour of stakeholders, learns and selects the behaviour with their associated objective value. The purpose of VRPSTW in the MAS model is to plan and implement delivery schedules of trucks for each freight carrier and neutral carrier. These two models are executed sequentially.

Encourage to join the joint delivery systems

Urban Logistics Association

Administrator

Residents

Legend

Fixed behavior agent

Decision making agent

Interaction

«ft Goods

« Information

Exhaust emissions

Ö Money

Fig. 1. Stakeholders' interaction within the MAS model

Learning-Model

Encouraging stakeholders to join JDS

Freight Carriers

Shop Owners

Delivery-Pickup

Neutral Carrier

Logistics Association

Residents

Make a complain

— Administrator

Information acquired by NC

Transportation profit

Environment impacts

Implementing logistic measures

VRPTW-Model

VRPSTW

Freight carrier 1

Freight carrier 2

Pickup and Delivery

Neutral carrier

Fig. 2. Proposed MAS model framework with vehicle routing and scheduling problem with time window

4. Multi-Agent System Model Framework

The Multi-Agent System (MAS) is a system composed of multiple interacting intelligent agents. MAS can be used to solve problems that are difficult or impossible for an individual agent or a monolithic system to solve. Intelligence may include some methodical, functional, procedural or algorithmic search, acquisition and processing approaches. Moreover, MAS is a useful methodology to examine the multi-objective nature of an urban logistics system and to study the behaviour of the stakeholders, who are affected by the freight policy measures. MAS consist of an environment with multiple autonomous agents with the ability to distinguish, perceive and take action while incorporating the interactions of other agents (Teo, Taniguchi and Qureshi, 2012). Additional information in MAS can be found in related sources (Weiss, 1999; Wooldridge, 2009).

4.1. Vehicle routing problem with time window (VRPTW)

VRPTW model, plans and implements delivery routing and schedules of trucks for each freight carrier. This paper includes the study of delivery and pickup activities from the shop owners in the shopping street by planning and implementing delivery routing and schedules of trucks for a neutral carrier (UDC truck operation). Likewise, this paper seeks to modify the MAS model framework for vehicle routing and scheduling problem with time window-forecasted (VRPTW-F) (Tamagawa, Taniguchi and Yamada, 2010) as shown in Fig. 2.

To determine the optimal solution by minimizing the total transport cost of freight carriers and neutral carriers, this research has applied the vehicle routing and scheduling problem with soft time windows (VRPSTW) model by Qureshi (2008) to study the delivery and pickup activities.

The model can be formulated as follows:

subject to

k£K (i,j)€A

xijk — 1

kEK jev

^ di ^ xijk < q

iEC j €V ^ x0jk = 1

^ xihk ~ ^ xhjk = 0 i£V

_| xi0k — 1

a!i< s'* < b'i

ai ^ sik < b'i

sik + tij - sjk < (l - xijk)Mijk

xuk e {0,1}

Vh E C.Vk E K

Vi E V,Vk E K Vi e V,Vk E K V(i,j) EA.VkEK (i,j) EA.VkEK

(8) (9)

The two decision variables in the VRPSTW are the service start time, sjk of truck k e K at vertex j e C, that will determine the arrival time at vertex j e C and travel cost of arc (i, j), and xtjk, where Xjk = 1 when arc (i, j) is used and xijk = 0 when arc (i, j) is not used in the solution. The objective function (1) minimizes the sum of delivery costs that consist of the fixed vehicle utilization cost, travel cost on arcs and the penalty costs. Constraint (2) ensures that each customer is serviced only once and constraint (3) makes sure that the load carried by the vehicle is within the limit of the vehicle's capacity. Constraints (4) and (6) determine that the vehicle shall start and end at

the depot while constraint (5) ensures that the vehicle entering vector h must also leave from vector h. Constraint (7) restricts the arrival time to be within the relaxed time window of at' and bt' and constraint (8) ensures that the service start time is within at and bi'. Constraint (9) shows that if a vehicle travels from i to j, the service at vector j can only start after service at vector i is completed. The last constraint, (10) is the integrality constraint, which completes the model formulation.

The problem described here is a NP-hard (Non-deterministic Polynomial-hard) combinatorial optimization problem. Thus, some heuristic algorithms are required to provide good and fast solutions for the MAS model. The model described here uses insertion heuristics to solve the VRPSTW.

4.2. Q-learning theory

Q-learning is a reinforcement learning technique that works by learning an action-value function that gives the expected utility of taking a given action in a given state and following a fixed policy thereafter. One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment. A recent variation called delayed Q-learning has shown significant improvements, bringing approximately correct learning (PAC) bounds to Markov decision processes (Alexander, Lihong, Eric, John and Michael, 2006). A typical learning algorithm for the freight carriers and shopping street association can be represented by equation (11).

Q(.st,at) ^ (1 - a)Q{st,at) + a [rStAt + YminQ(st+1,at+1)] (11)

where ,

Q(st, at)Q(st, at) : Q-value in state t due to action in state t. Q(st+1, at+i)Q(st+i> at+1) : Q-value in state t+1 of all actions. y : discount rate for agent (0 < y < 1).

a : learning rate for agent (0 < a < 1).

rStAt : immediate reward in state t due to action in state t.

The learning rate of 1 means that the agent will consider the most current information while 0 means the agent does not learn. A discount rate set at 1 means that the agents will consider the long term reward while 0 means that the agents are only concerned with the current rewards.

Truck emissions are considered in the city logistic scheme to evaluate the benefits of using the UDC. Besides this, the joint delivery and parking management will be considered as policy measures. The oxides of nitrogen (NOx), carbon di-oxide (CO2) and suspended particulate material (SPM) equations are used to estimate truck emissions.

The oxides of nitrogen (NOx), carbon di-oxide (CO2) emission and suspended particulate material (SPM) are estimated using equation (12), (13) and (14) respectively (NILIM, 2003) assuming delivery truck vehicles use diesel fuel.

N0T = 1:

0.000213v?j - 0.0246vu

16.258\

COn = h: (278.448 + 0.048059^,- - 5.1227v;;

SPM = lij (0.03442 - 0.000039391^5

0.0036777Vi¡

where,

N0X CO2 SPM hi

1.2754\ vii )

Expected nitrogen oxide emission in grams. Expected carbon di-oxide emission in grams. Expected suspended particulate material in grams. Length of road link between nodes i and j in kilometres.

Vij : Speed of vehicle travelling on road link between nodes i and j in kilometres per hour

4.3. Stakeholders associated with urban freight transport

In a multi-agent model, stakeholders have their own objectives as follows; Freight Carriers

Objective: Minimize operation cost.

(eg. Less truck used etc.)

Behaviour: Propose the fee for transporting goods to shop owners.

Shop Owners

Objective: Minimize delivery cost.

Behaviour: To participate or withdraw from the joint delivery system.

Residents

Objective: Minimize the emissions by trucks.

Behaviour: Complain to the administrator when emissions in their area

exceed the environmental limit.

Administrator

Objective: Minimize the number of areas where

the residents complain about emissions.

Behaviour: Encourage freight carriers and shop owners to use UDC

by issuing parking space regulation limitation.

Neutral carriers

Objective: Maximize the profit of delivery goods.

Behaviour: Propose the fee for transporting goods to shop owners

without delay.

Shopping Street Association

Objective: Maximize the number using the UDC.

Behaviour: Encourage freight carriers and shop owners to use the UDC

with subsidies.

5. Experiment Setup

Fig. 3 shows the testroad network used in this study. The four freight carriers are named as carriers A, B, C and D and are located at nodes 2, 11, 15 and 22 respectively. Nodes 9, 14 and 19 are the locations of shop owners whilst the rest of the nodes represent the residents. This network is assumed to be an urban area with congested traffic conditions and crowded shopping streets. The four freight carriers have their own depot. The assumption and city logistic measure policies that are used in this study are described in Table 1 and Table 2 respectively. The MAS model is iterated for 360 days, which is equivalent to a year. The experiment flow of without/ with the UDC operations are as follows;

Fig. 3. Test road network

5.1. Without UDC case

(Step 1) Freight carriers deliver goods to residents.

(Step 2) Freight carriers go to the shop owners to deliver and pickup goods that are needed for delivery to residents in the next step.

(Step 3) Repeat step 1 with the amount of pickup goods in step 2 for the next day.

5.2. With UDC case

(Step 1) All freight carriers deliver goods to the UDC.

(Step 2) Divide the distribution activity into two scenarios. Firstly, the neutral carrier delivers and pickups goods to/from shop owners. Secondly, other trucks from neutral carriers deliver goods to the residents, which included the goods picked up from shop owners.

(Step 3) Repeat step 1 and 2 for the next day.

Table 1. Modelling assumptions

Modelling assumption

General assumption

Service time for delivery is from 8 AM. to 8 PM.

There is only one type of truck.

There is only one type of goods.

The quantities of delivery and pickup goods are dynamically assigned throughout the year. The time window of delivery and pickup goods is dynamically assigned throughout the year.

Access to the UDC is closed to the freight carriers.

The UDC can have an early delivery, fixed time deliveries or full truck delivery scheme. Various UDC usage charges are "free of charge", 100 yen/parcel and 150 yen/parcel.

Freight carriers and neutral carrier trucks Vehicular costs are fixed.

Truck capacity is 130 parcels.

Service time window ranges between 15 to 35 minutes. Freight carriers travel with an average velocity at 30 km/hr. Penalty charge for early delivery is 10 yen/minute. Penalty charge for delay delivery is 50 yen/minute.

Table 2. Condition of Policy Measures

Policy Measure Affected Agent Condition

Subsidies from Shopping Street Association - Neutral carrier - Freight carriers - Subsidies 50 yen and 75 yen from Shopping Street Association.

Parking Management - Neutral carrier - Neutral carrier has the first priority of parking space without any charge.

- Freight Carriers - Parking truck at normal car parking with 100 yen charge per 10 minutes and continuation of delivery to shop owners on foot with additional service charge 100 yen per minute.

- Administrator

6. Result and Discussion

In this paper, the distribution events included three activities to evaluate the benefit of the city logistics policies:

Activity 1: The delivery activity serviced by the freight carriers requires the carriers to deliver directly to their customers.

Activity 2: The delivery activity serviced by the neutral carrier with various UDC usage charge and car parking

management policy implemented by the administrator. Activity 3: The delivery activity serviced by the neutral carrier with supported subsidies from the shopping street association and car parking management policy implemented by the administrator.

Fig. 4 shows the trend of frequency of UDC usage declining when the UDC charge increases, which led to more occurrence of freight carriers delivering the goods directly to their customers. However, the incremental subsidy rate by the shopping street association at each UDC charging rate resulted in the increase of UDC usage. It shows that the subsidy tends to encourage the freight carriers to use the UDC.

Figs. 5, 6 and 7 show the results of decremental emissions of nitrogen oxide (NOx), Carbon dioxide (CO2) and suspended particulate matter (SPM) when the UDC and subsidies are implemented together with the car parking management. It is shown from the graphs that as the unit rates for each parcel reduce, the emissions reduced accordingly with the lowest emission recorded when there is no charge for using the UDC. Besides, the subsidy rate by the shopping street association is observed to reduce the emission further and at some point, the subsidy is seen to reduce the emission at a higher rate, for example at the point between zero to 50 yen when there is no charge for the UDC usage, and between 50 yen and 75 yen when the UDC usage is charged at 150 yen per parcel.

The performance of the freight carriers' cost and the number of trucks used are shown in Figs. 8, 9 and 10 with 0 yen, 50 yen and 75 yen subsidy respectively. The cost to carriers may have risen when the unit price of delivering a parcel increased but the overall cost is still lower than the base case, when no policies are implemented. The increase in subsidy also helped to reduce the carriers' cost for each category of unit rate per parcel. For example, when it is free to use the UDC, the cost of carriers reduced from 87 million yen to about 62 million yen as the subsidy increased. This particular example also shows that the reduction rate is higher when it is free to use the

UDC as compared to the case when the UDC is charged with 100 yen per parcel and 150 yen per parcel. In the comparison of truck use, the subsidies of 50 yen and 75 yen can maintain a lower truck usage as compared to the base case. This is in contrast with the implementation of a single policy of UDC charges without subsidy as shown in Fig. 8, where the trucks may increase as the price of the UDC fee increases. Based on the results observed, we can see the potential of the joint schemes of the UDC and the subsidy from the shopping street association.

Fig. 4. Frequency comparisons of UDC usages

1,200 1,000 800 600 400 200 0

Subsidy

^ ___ _.

0yen 50yen 75yen

< Base Case

B- — Free UDC Usage

— * - 100yen/parcel UDC fee

••••*•■• 150yen/parcel UDC fee

Fig. 5. NOx emission against subsidies and UDC charges

u o ht n

250,000

200,000

150,000

S 100,000

50,000 0

Subsidy

• Base Case

— Free UDC Usage

* - 100yen/parcel UDC

•*••• 150yen/parcel UDC fee

Fig. 6. CO2 emission against subsidies and UDC charges

40 35 30 25 20 15 10 5 0

Subsidy

> Base Case

— Free UDC Usage

* - 100yen/parcel UDC fee

•X — 150yen/parcel UDC fee

Fig. 7. SPM emission against subsidies and UDC charges.

¥140M

¥120M

¥100M

^ ¥80M

¥0M UDC Fee

Base Case Free

Freight Carrier Cost

100 yen/parcel

150 yen/parcel "Truck Use

7,000 6,000 5,000 4,000 3,000 2,000 1,000

Fig. 8. Performance comparison of freight carriers' cost and truck used without subsidy

¥140M

¥120M

¥100M

•S ¥60M

¥0M UDC Fee

Base Case

100 yen/parcel

Freight Carrier Cost

150 yen/parcel Truck Use

3,000 o

Fig. 9. Performance comparison of freight carriers' cost and truck used with 50 yen subsidy

¥140M

¥120M

¥100M

S3 ¥80M eirr a

js ¥60M

¥0M UDC Fee

Base Case

100 yen/parcel

Freight Carrier Cost

150 yen/parcel Truck Use

3,000 o

Fig. 10. Performance comparisons of freight carriers' cost and truck used with 75 yen subsidy

7. Conclusion

In this paper, the multi-agent system model was used as a methodology to evaluate the benefits of using the urban distribution centre, which comprised of transportation cost and environmental issues,. The findings of operating cost reduction and minimized environmental impact for implementing UDC is encouraging. The urban freight emission reduction is achieved from the replacement of individual delivery to consolidated delivery with the presence of a UDC. In addition, the subsidy from the shopping street association and by applying parking management as the additional supporting scheme helps to fine tune the implementation of the UDC for better results. Future studies will continue to evaluate other schemes like spilt deliveries of neutral carriers and apply these to a real network.

References

Alexander, L. S., Lihong, L., Eric, W., John, L., and Michael, L. L. (2006). Pac model-free reinforcement learning, In Proc. 23nd ICML 2006, 881-888.

Antal, V. K. (2010). Multi-agent model for the urban distribution centre: Scenario search and dynamic urban distribution centre pricing to find a

positive business case, Master Thesis, Delft University of Technology. Dablanc, L. (2007). Goods transport in large European cities: Difficult to organize, difficult to modernize, Journal of Transportation Research Part A 41, 280-285.

NILIM (2003). : Qualitative appraisal index calculations used for basic unit for computation of CO2, NOx and SPM (in Japanese).

Quak, H. and Koster, M. (2009). Delivering goods in urban areas: How to deal with urban policy restrictions and the environment, Journal

Transportation Science, 43(2), 221-227. Qureshi, A. G. (2008). A column generation based exact solution for vehicle routing and scheduling problem with soft time windows, Doctoral Thesis, Kyoto University.

Tamagawa, D., Taniguchi, E. and Yamada, T. (2010). Evaluating city logistics measure using multi-agent model, Procedia Social and

Behavioral Sciences., 2, 6002-6012. Taniguchi, E., Yamada, T. and Okamoto, M. (2007). Multi-agent modelling for evaluating dynamic vehicle routing and scheduling systems,

Journal of the Eastern Asia Society for Transportation Studies, 7, 933-948. Teo, J. S. E, Taniguchi, E. and Qureshi, A. G. (2012). Evaluating city logistics measure in e-commerce with multi-agent systems, Procedia

Social and Behavioral Sciences, 39, 349-359. van Duin, J. H. R., Kolck, A., Anand, N., Tavasszy, L. and Taniguchi, E. (2012). Towards an agent-based modelling approach for the

evaluation of dynamic usage of urban distribution centres, Procedia - Social and Behavioral Sciences, 39, 333-348. Vita, G. and Janis, G. (2005). Technologies and multi-agent system architectures for transportation and logistics support: An overview.

International Conference on Computer Systems and Technologies - CompSysTech' 2005. Weiss, G. (1999). Multiagent Systems: A modern approach to distributed artificial intelligence. Cambridge, Massachusetts: The MIT Press. Wooldridge, M. (2009). An introduction to multiagent systems - 2nd edition, West Sussex, United Kingdom : John Wiley and Sons.