Application of Agent Technology in Warehousing - Toepassing van agent technologie in distributiecentra

Delft University of Technology

FACULTY MECHANICAL, MARITIME AND MATERIALS ENGINEERING

Department Marine and Transport Technology Mekelweg 2 2628 CD Delft the Netherlands Phone +31 (0)15-2782889 Fax +31 (0)15-2781397 www.mtt.tudelft.nl

This report consists of 53 pages and 0 appendices. It may only be reproduced literally and as a whole. For commercial purposes only with written authorization of Delft University of Technology. Requests for consult are only taken into consideration under the condition that the applicant denies all legal rights on liabilities concerning the contents of the advice.

Specialization: Transport Engineering and Logistics Report number: 2016.TEL.8079

Title: Application of Agent Technology

in Warehousing

Author: L. van Noort

Title (in Dutch) Toepassing van agent technologie in distributiecentra

Assignment: literature Confidential: no

Supervisor: Dr.ir. Y. Pang Date: January 16, 2017

T

U

Delft

FACULTY OF MECHANICAL, MARITIME AND MATERIALS ENGINEERING

Delft University of Technology Department of Marine and Transport Technology

Mekelweg 2 2628 CD Delft the Netherlands Phone +31 (0)15-2782889 Fax +31 (0)15-2781397 www.mtt.tudelft.nl

Student: L. van Noort Supervisor: Y.Pang Specialization: TEL Creditpoints (EC): 10

Assignment type: Literature Assignment Report number: 2017.TEL.8079 Confidential: No

Subiect: Application of Agent Technology in Warehousing

Agent technology offers the approach to design, build, monitor and control complex distributed

system. Besides modelling distributed systems as software programs exchanging data and commands, agent technologies create autonomous decision makers which communicate their explorations,

negotiate their preferences and coordinate their intensions to achieve the operational goals on both individual and system levels.

The application of agent technology is booming nowadays in the field of transport and logistics. This assignment is to provide a survey of the state of the art agent applications in warehousing. The survey of this literature assignment should cover the following:

• to describe the functions and processes in warehousing operations • to understand the concepts of agent and agent systems

• to investigate the characteristics, general architectures, basic interaction protocols and intelligent abilities of agents in the domain of warehousing

• to explore the applications of agent technology in warehousing

• to demonstrate the intelligent control for warehousing by means of agent systems

This report should be arranged in such a way that all data is structurally presented in graphs, tables, and lists with belonging descriptions and explanations in text.

The report should comply with the guidelines of the section. Details can be found on the website. The mentor.

Application of Agent Technology in Warehousing

Laura van Noort

January 16, 2017

Abstract

This report describes the implementation of agent technology in warehous-ing. The main processes in warehouses are receiving, put-away, storage, order picking and shipping. The dynamic environment and all the different processes going on at the same time, require a system that consists of components that can function autonomously and can can adapt easily as they need to react to real-time information. Agents are autonomous and adaptive software systems which receive percepts and can perform actions on their environment. Multi-ple agents can work together in multi-agent systems. Types of agents that are often used in warehouses are order agents, planning agents, scheduling agents, control agents and AGV agents. They each fulfill a certain warehouse task. These agents work together in systems. There is not one specific way of design-ing these systems. A lot of different structures are used. In some cases there is a hierarchical structure where higher level agents controls the lower level agents, while in other systems all agents communicate with each other. One of the benefits compared to existing systems is that agent systems are decen-tralized, so there is not one entity that has all the information and can cause a bottleneck in the system. Another huge benefit is that agents are adaptive and can easily respond to a dynamic environment found in warehouses.

List of Figures

2.1 The number of shipments without and with a warehouse. The black lines

represent the shipments [1]. . . 4

2.2 Different components in warehouse process. (Source: [2]) . . . 5

2.3 Different ways of product flow through the warehouse. (Source: [3]) . . . . 5

2.4 Different ways of order picking. (Source: [4]) . . . 8

2.5 Miniload system (Source: [5]) . . . 9

2.6 Carousel system (Source: [6]). . . 9

2.7 Vertical lift module (Source: [7]) . . . 9

2.8 AGVs used in warehouses. (Source: [8]) . . . 10

2.9 KIVA system for warehouses [9]. . . 11

2.10 The different layers when using warehouse management and warehouse control systems. (Source: [2]). . . 15

2.11 General model of a warehouse system. This system can be divided into two subsystems [10]. . . 16

3.1 Schematic drawing of an intelligent agent. It receives percepts from its environment by sensors and can perform actions on the environment with the help of effectors (actuators). Source: [11] . . . 18

3.2 Examples of PAGE [11]. The different percepts, actions and goals are given for different agent types. . . 19

3.3 Layered architecture of an agent (Source: [12]) . . . 25

3.4 Different control architectures. (a) shows a single layered control structure. (b) shows a hierarchical control structure. The agent in the upper layer has direct control over the lower level agents. The dotted lines show the possible ways of communication of the agents with each other. (Source: [13]) . . . 27

3.5 Holonic Architecture. Agent 7 (A7) is the head of a superholon consisting of two smaller holons (A1, A2 ,A3; and A4, A5, A6) (Source: [14]) . . . . 28

3.6 Coalition Architecture. (Source: [14]) . . . 28

3.7 Team Architecture. (Source: [14]) . . . 28

4.1 Determining the Priority based on Distance, Waiting Time and Request Ration. (Source: [15]) . . . 33

4.2 The fuzzy sets of the variables distance, waiting time and request ratio. (Source: [15]) . . . 33

4.3 Determining the priority based on distance, waiting time and request ratio. (Source: [15]) . . . 34

4.4 Overview of a pick zone. (Source: [16]) . . . 35

4.5 Negotiation gantry agents on conveyor speed. (Source: [16]) . . . 35

4.6 TSP with pick locations and drop buffer locations. (Source: [17]) . . . 37

4.7 Mathematical model of the TSP. (Source: [17]) . . . 38

4.8 A problem in the x-coordinate sequence occurs if the pick locations are situated close together (Source: [17]) . . . 38

4.9 Six generic agents for warehouse control and the three levels of operation. (Source: [18]) . . . 39

4.10 Different agents and sub-agents used [10]. . . 40

4.11 System architecture. The process of sending materials to the customer [10]. 40 4.12 Schematic overview of the system architecture [16]. . . 42

4.13 Schematic overview of the system architecture [16]. . . 43

List of Tables

3.1 Agent Characteristics . . . 22

3.2 Different sorts of agents . . . 23

4.1 General overview of research done on agent technology in warehousing . . 31

Contents

1 Introduction 1 2 Warehousing 3 2.1 Warehouses . . . 3 2.1.1 Function . . . 3 2.1.2 Processes . . . 5 2.2 Equipment . . . 9 2.3 Management of Warehouses . . . 11 2.3.1 Planning . . . 11 2.3.2 Control . . . 12 2.3.3 Current Systems . . . 15 2.4 Shortcomings . . . 15 2.5 Requirements . . . 16 3 Intelligent Agents 18 3.1 What is an Agent? . . . 18 3.2 Characteristics . . . 20 3.3 Classes of Agents . . . 23 3.4 Decision Making . . . 24 3.5 Agent Systems . . . 25 3.5.1 Structure . . . 26 3.5.2 Control Architecture . . . 27 3.5.3 Interaction . . . 28 4 Application in Warehousing 30 4.1 Lower Level Agents . . . 32

4.1.1 AGV Agents . . . 32

4.1.2 AS/RS Agents . . . 34

4.2 Higher Level Agents . . . 36

4.2.1 Planning Agents . . . 36 4.2.2 Control Agents . . . 36 4.3 Set of Agents . . . 38 4.4 Control Architecture . . . 42 4.5 Interaction . . . 44 4.6 Summery . . . 44 5 Conclusion 48 Bibliography 49

1

Introduction

Warehouses play an essential role in logistics and do not only function as storage space, but also function as buffer for seasonal products or to improve the shipment of products. With the rise of e-commerce businesses the focus on efficiency in ware-housing increases. Especially nowadays with the promise of same day delivery by several online shops warehouses require accurate and fast handling of products to keep the customer happy. Performance of the warehouse is key. This causes a great strain on the warehouses where the goods are stored. Recently it was revealed that the working conditions at the warehouses of a well-known online shop were so bad that almost every day an ambulance had to be called [20]. The biggest strain lies on the order pickers and the people working in the packing areas. These are two of the most time consuming processes of the warehouse. In a warehouse different processes happen at the same time. During the whole day orders can come in, which causes a constantly changing environment. The right products have to be picked from the right racks and have to be brought to the packing areas to fulfill the orders. From the packing areas the boxes have to go to the right shipping dock to transport the packages to the customers. All these processes are handled by a large number of in-dividuals (machines or humans). To make sure all these processes happen in a right order and are well organized, there needs to be an overall control or management system that has an overview of all the processes happening and of all the incoming orders that need to be fulfilled. To improve the working conditions and performance at the same time one can opt for automating the warehouse processes.

Because the environment in the warehouse is constantly changing, warehouses re-quire an adaptive system that handles the processes real-time. All the different processes happening on the warehouse floor at the same time makes that these pro-cesses have to be executed independent from each other and components should be able to act autonomously. At the same time communication is an important part of the warehouse to organize all the processes. A warehouse can be divided into two layers; the executive layer and the control/management layer. The executive layer is the lower level and consists of the equipment used in the warehouse, e.g. AGVs, conveyor belts and picking machines. The control/management layer has an overview of all the processes and actions happening in the warehouse. The different pieces of equipment have to communicate with each other for the improvement of the functioning of the individual equipment. For example AGVs not running in to each other or blocking each others paths. Considering the management it is also important that this layer can communicate with the lower level layer, for example by giving directions or even to overrule the actions of the lower level. Because of the overview of all the processes and incoming orders, this layer can schedule the actions of the lower layer and therefore set goals for the lower level. In other words, this system needs to instruct the executive layer.

With technology getting more and more advanced, an increasing amount of pro-cesses are getting automated. The possibilities of ICT are tremendous. Especially the field of logistics is ideally suited for automation. A lot of different warehouse automation systems exist already, but most of these systems are centralized and are not very flexible. One of the possibilities of using ICT to automate a warehouse is

agent technology.

Agent technology offers a new approach to automating warehouses. Because agents are autonomous and can adapt easily, the automation of warehousing could benefit enormously from these systems. In the broader field of supply chain and in the field of manufacturing a lot of research has been done already on the implementation of agent technology [21]. However in the field of warehousing it is still a fairly new approach.

The goal of this paper is to give an overview of the work done on the implementation of agent systems in warehousing. To accomplish this, it is important to:

• describe the different functions and processes in warehouses • understand the concepts of agents and agent systems

• investigate the characteristics, general architectures, basic interaction proto-cols and intelligent abilities of agents in the domain of warehousing

• explore the application of agent technology in warehousing

• demonstrate the intelligent control for warehousing by means of agent systems

This report focuses on the control of warehouse components and the structure of the multi-agent systems. The software design of these agents lies outside the scope of this assignment.

To give an overview on the implementation of agent technology in warehousing chapter two first describes the functions and processes in warehouses. Also the existing equipment and management and control of the warehouses will be described in this chapter. Chapter three describes what exactly an agent is and shows how agents can be used to build intelligent systems consisting of multiple agents. Chapter four will give an overview of agent use in the field warehousing. Based on these chapters a conclusion will be given in chapter five.

2

Warehousing

Warehouses play an essential role in logistics and are used by i.a. manufacturers, wholesalers, customs and e-commerce businesses. Goods coming from suppliers need to be stored, to wait for further transport to the customer or to wait for further processing. This takes up space and time, which are both costly. For this reason efficiency is important. A warehouse can be divided in different levels. The lower level can also be seen as the execution level which consists of the equipment and workers. The higher level consists of the management and control of the warehouse. This chapter describes the general functions and processes within a warehouse and the equipment used in these processes (lower level). Also the management and control of the warehouses will be discussed (higher level).

2.1

Warehouses

The earliest form of storing goods are storage pits made by farmers to store seeds and food. These are still far from the warehouse facilities that exist nowadays. The first warehouses that come closer are the ones that were build in the Roman Empire. These were mainly used to store goods as grain and olive oil. Rome had about 300 warehouses and these warehouses received over 400.000 metric tons per year in bulk. It was only during the industrial revolution that the need for warehouses became real high, but it was not until the 20th century that major innovations took place. One of the biggest improvements was the standardization of pallets in 1930. Before that time crates, boxes and barrels always had to be moved one at the time, but with the introduction of a standardized pallet it was able to move multiple goods stacked on one pallet. This also lead to the development of forklifts. Another big improvement was the invention of the the bar code. This was a big step that made the automation of warehouses possible. Modern Day warehouses get more and more advanced. With the rise of e-commerce, so called fulfillment centers have been build, which is nothing more than a modern word for warehouses. The bigger warehouses of companies like Amazon or Zalando cover over 100.000 square meters.

2.1.1 Function

This section is based on the work of Bartholdi [1]. Where in the past warehouses were mainly used for storage of products, today there is a lot more to warehousing. Bartholdi describes the different functions of a warehouse. Warehouses can also function as buffers. For example with seasonal products. Some companies as toy stores are a lot busier during the months November and December. During this time the warehouses of stores like Toys R Us have to do a lot of shipments. The other part of the year they spend on rebuilding their inventory, so the next year they can handle all the orders again during the winter months. Instead of building a inventory for busier times it also works the other way around. The buffer function can be used when demand collapses for it can store inventory from the market. Also with the variance of prices a company might decide to order a certain product in bulk to save a lot of money. This product needs to be stored somewhere. Again a warehouse can function as a buffer in this case. Another function of a warehouse is to improve the shipment of products. Whenever an ordered product has to come all the way from China, it is prone to delay caused by for example congestion, weather

Figure 2.1: The number of shipments without and with a warehouse. The black lines represent the shipments [1].

or bureaucracy. During the process of transport delay in one part of the journey can cause even more delay on other parts. If a product can be held closer to the customer, delay is less likely to happen. Also the warehouse can be used to con-solidate shipments into large shipments. If there is a fixed cost to a shipment it is more efficient to send a lot of products at the same time. The warehouse collects all the shipments from the different suppliers and sends them in large shipments to the stores. Figure2.1 shows that there is a severe decrease in shipments when using a warehouse.

The functions within a warehouse also depend on the warehouse type. A retail distribution center supplies retail stores. Most shipments are regularly scheduled on fixed days. Those warehouses often serve a lot of stores, so the flow through the warehouses is high. A service parts distribution center holds spare parts for capital equipment. In contrast to a retail distribution center it is hard to forecast the demand for certain parts. For a service parts distribution center it is impor-tant to have a large stock. Product flow in these warehouses is not as high as in retail distribution centers and especially emergency orders are expensive to handle. A catalog fulfillment or e-commerce distribution center is a rapidly growing type of warehouse. It receives respectively small orders placed by customers via the internet (or sometimes by phone or mail), but the number of orders coming in every day is very high. This means that the product flow is large. Because most of these ware-houses want to ship the items to the customer as soon as possible (some even offer same day delivery), the orders need to be picked as soon as they come in. 3PL ware-houses are wareware-houses used for outsourcing. Whenever a company does not have the capacity to store its own good, the goods can be stored at a 3PL warehouse. These warehouses often serve more than one company. A perishables warehouse handles perishable goods. These can be for example food, flowers and vaccines. The time these products stay in the warehouse is not very long. Spacing is important in these warehouses, because control of the warehouse environment, temperature and humidity, is important.

only stays in a warehouse for a shorter amount of time (a couple of hours at most). This is often done to reduce transportation costs. Items are spared, so trucks can be fully loaded. Crossdocks do not have a lot of storage space. They do however have a lot of material-handling equipment to move the products from the unloading area to the loading area. The benefits are less inventory and less labour for there is no order picking process.

2.1.2 Processes

Figure 2.2: Different components in warehouse process. (Source: [2])

Warehouses can have differ-ent functions, but the over-all processes stay the same (except for cross docking warehouses). Figure 2.2

shows the five processes in a warehouse. When a load comes in, it needs to be picked up from the unload-ing area, to be transported to the shelf were it will be stored. Receiving and

putting away are the inbound processes of the warehouse. At the same time, stored goods need to be transported from the shelf to the the place where they are picked up for further processing or to go to the customer. Picking and shipping are the outbound processes of the warehouse. In between these two processes there is the storage process. Goods can be stored as bulk, mostly for a longer period of time, or in smaller amounts, so they can be easily picked by the order pickers. Typically loads enter in a different composition than when they leave the warehouse as loads coming in on pallets can leave in boxes or as single items. There are a lot of different ways loads can be handled. The possible ways of flow of products through the ware-house are shown in figure 2.3. Products normally are delivered in large packages, mostly packed on pallets. The content of the pallets can be separated into smaller packages as cartons or even to single items (broken case picking). Loads can be stored in all three forms.

Receiving Process

In the receiving process the scheduling of the arrival of trucks is an important aspect. Trucks need to be assigned to docks and loads need to be unloaded, so planning of labor and material handling equipment is another important aspect. Products that are unloaded will be staged to be put away. The goods arriving into the warehouse need to be registered for a good product management (this is mostly done by scan-ning the bar codes or by the use of RFID chips). Also the goods need to be inspected before they can be put away (checks on damage, quantity and descriptions will be done). Before the goods can be put away, some of them need to be separated into smaller amounts or in case of different goods arriving on one pallet, they need to be sorted.

Put-Away

In this process it is important to determine the place ware the goods must be stored. The performance of the put away process as well as the picking process will both benefit from a good storage location. Also the management of storage locations plays a role in this process. At all times it must be known which storage locations are available and which are not, what the size of the storage location is and what weight it can bear. At the same time it must be known which goods are stored at what location, so the same goods are stored in the same place. Once a product has been stored it is important that the location of this product will be registered. Putting loads away in storage is mostly done in larger quantities and the distances to the location of storage are usually larger than for the retrieving of products. For this reason sometimes different equipment is used for put-away compared to order picking.

Storage Process

Items can be stored in different sizes, they can be stored on pallets or in smaller sizes so they are easy to handle for order pickers. Most storage locations are di-vided in different zones, this is called zoning. Each zone can have its own storage equipment and order pickers are assigned to just one zone, so the distances they have to travel are smaller [22]. An important action in storage is the replenishment of goods. Most items are stored on pallets on higher racks (reserve storage), while the smaller amounts ready for order picking are stored on easily accessible racks (forward storage). Replenishment is the transfer of items from reserve storage to forward storage [23]. Gu [24] describes three important decisions that should be made in the storage process. The first two have to do with control of the stock. These questions are: "how much inventory should be kept in the warehouse?" and "how frequently should the inventory be replenished?". These questions should be answered using historical data. The third question is: "where should the load be stored in the warehouse?". This question has to do with assigning loads to storage departments and inventory move and is important for efficient order picking.

Order Picking Process

The order picking process is the most time consuming process of all warehouse pro-cesses. When an order has been received, stored items need to be taken from storage to prepare for shipping. Order picking can be done in two different ways [3]. The first is picker to parts where the picker has to move to the right rack to pick up

the goods. This is mostly done using forklifts. The second way is parts to picker. In this case the parts move to the picker and these parts are picked at a station. This is mostly done by an automated storage retrieval system (AS/RS). This system uses aisle bound cranes to retrieve the loads. Other systems that can be used are mini-load, modular vertical lift modules (VLM) or carousels. These systems will further be discussed in detail in section2.2.

Picker to parts order picking is done using pick-lines. These are instructions telling order pickers what to pick where and in what quantity. Order picking can be divided in four smaller processes [1]:

• Traveling • Searching • Extracting

• Paperwork and other activities

Traveling takes 55% of the time, while searching takes 15% of the time. Extracting takes 10% of the time and paperwork and other activities 20%. There are different lines for carton picking and picking single items. If a customer ordered 15 pieces of one item, instead of picking 15 pieces, it can also be decided to pick one carton (in this example containing 12 pieces) and 3 pieces as single items. This reduces the extracting time, but might increase travel time. The picking sequence can reduce travel time tremendously and can make the order picking more efficient. If picking locations are planned in a way that the order picker has to go from on side of the warehouse to the other side and back again to pick another item then this is not efficient. Therefor it is important to make sure all picking locations are in such an order that the order picker would naturally encounter them while moving through the warehouse.

Put systems are placed between parts-to-picker and picker-to-parts systems and the shipping zone. When the items are retrieved by one of the retrieval systems a bin with all the picked items is offered to a picker. The picker then distributes the items over the right boxes that will be sent to the customer. Using a put system can result in 500 packages per picker per hour [4]. These systems are mostly used in warehouses that need to process many small orders as for example Amazon.

It can be said that the higher the pick density, the more efficient the order picking [1]. This can be improved by storing the items together that are ordered most. Another way to do this is by picking more than just one order in a picking tour. This means that the pieces must be sorted. The sorting can be done during picking (sort-while-pick ) or after the picking has been finished (pick-and-sort ). Another option is zoning. The warehouse is divided into different zones. Order pickers are assigned to one particular zone. This can mean that more than just one order picker is working on an order. The advantages are that order pickers need to travel smaller distances, that there is a reduced traffic congestion and that pickers get to know the zone quicker, so time spent on searching for a product gets reduced. The disadvantage is that the pieces belonging to the order need to be aggregated

again. This can be done by using the pick-and-pass system in which one order picker first picks all the orders in his zone and than passes on the list to the order picker in the next zone. Another approach is parallel picking in which order pickers in different zones work on the same order. When all pieces are retrieved the order gets assembled at the end. This is also called synchronized picking. An overview of the order picking process can be seen in figure2.4.

Figure 2.4: Different ways of order picking. (Source: [4])

The last action to be taken in the order picking process is the packing of the goods so they can be shipped. The package needs to be registered to show it is available for shipping.

Shipping Process

The shipping process consists of loading the packages into the truck. If products need to be loaded in a certain order, they will be staged before loading. Otherwise the loads can go straight into the container. Whenever the truck leaves this will be scanned, so the departure is registered.

Rouwenhorst [23] distinguishes two different warehouses with a focus on different processes; distribution warehouses and production warehouses. Distribution ware-houses store products and fulfill customer orders. The number of different products is high, while the number of products in one order are typically small. For this reason distribution warehouses focus often on the optimization of the order picking process. Production warehouses however store raw materials, work-in-processes and finished products. These are typically stored for a longer period of time. Therefore production warehouses focus more on the storage capacity of goods.

2.2

Equipment

To improve the processes in the warehouses different types of equipment can be used either to support the order picker or to fulfill the process completely automated. Different equipment exists for different processes. For storage there exist many different kinds of racks and storage units in which products might be stored, think of pallets, carton boxes or plastic boxes. This report will focus on the racks that are also used as retrieval system. As mentioned in the previous section there are several automated storage/retrieval systems. The most common used device is the miniload (figure2.5). In this device all loads are stored horizontally on high shelves. In between these shelves a robotic device is placed that can move vertically as well as horizontally to pick up the loads. After a load has been picked up from the rack it can be placed on a conveyor to be moved to the order picker. After the order picker has taken the pieces he needed the load is transferred back to the shelf. The aisles formed by the racks can be extremely narrow, so a system like this can save a lot of space

Figure 2.5: Miniload system (Source: [5])

Figure 2.6: Carousel system (Source: [6])

Figure 2.7: Vertical lift module (Source: [7])

The carousel system contains shelves that can rotate, as is shown in figure 2.6. In-stead of walking through the aisles, the loads from the aisles come to the order picker. At each picking station where the order picker is working multiple carousels are placed, so the order picker can pick from different carousels at the same time. After an order has been completed, the packages can be placed on a conveyor that takes them to the shipping area. Carousels can be placed side by side, because the products come to the order picker and there is no need to have an aisle be between them. The restocking of the carousel also has to be done by the order picker. With the vertical lift module (figure 2.7) again picking and packing takes place in the same area. It consists of high racks with a lifting device in between. This device picks up a tray from a shelf and places it at the receiving window. Whenever the order picker has picked the right items, the lifting device places the tray back on its shelf.

The conveyor system can have its own sorter. A common sorter is a push sorter which pushes the carton of the conveyor at the right spot. In picker to part systems the most commonly used equipment are order picking trucks as shown in figure2.8. These come in all sorts and sizes. Some need to be pushed manually, while others are motorized. The order picking trucks need to drive from the packing area to the

right rack where the item is stored, pick it up and return to the packing area.

For automated warehouses not using an automated storage/retrieval system, the alternative is using automated guided vehicles (AGVs) for order picking. Figure2.8

shows an example of AGVs used in a warehouse environment. The advantage of a warehouse is that the AGVs are used indoors and are less vulnerable for wear. An AGV system can be divided in different parts: the vehicles, the transportation network, the physical interface between the shipping and storage system and the control system [25]. The AGV can move along fixed paths, determined by wires or markings on the floor or can operate without these guide paths. The system has pick up points and delivery points. When an order is released to a certain AGV, it drives to the pick up point to pick up the load and then moves to the delivery point to deliver the load. In some cases a due time is given to the AGV, so the load will be delivered within a certain time.

Figure 2.8: AGVs used in warehouses. (Source: [8])

The flow paths AGVs can move along can be seen as a network consisting of arcs (the fixed guide path) and nodes (corners of aisles, intersections, pick up points and delivery points. In most cases the arcs are directed and the AGV can only travel in a certain direction. In unidirectional arcs the AGV can only travel one way, while in bidirectional arcs the AGV can travel both ways. The use of bidirectional arcs en-sures shorter travel paths, but are also harder to control, because an opposing AGV can cause a blockage and can even lead to a deadlock or collision. In warehouses with wide aisles it is possible to make multiple lane guide paths, in which opposing traffic is in a separate lane. In one warehouse network a mix of different paths can be used [25].

Another important aspect of AGVs is the avoidance of collisions and deadlocks. To avoid collisions, sensors as laser scanners or cameras can be used, so approaching

AGVs can detect each other. When a collision has been avoided still a deadlock can occur as both vehicles are standing in each others way. If deadlocks occur in a system, the performance will decrease [25].

A specialized AGV system for warehouses is the KIVA system [9] (figure2.9). Small robots drive autonomously from the storage area to the picking and packing stations carrying inventory pods. At the stations an order picker collects the needed items to prepare the package. When replenishment is needed, they also take the pods to replenishment locations. The control of the AGV systems will be discussed in section2.3.2. Other equipment that is necessary for a warehouse system are bar code scanners or RFID scanners. This depends on the system used in the warehouse.

Figure 2.9: KIVA system for warehouses [9].

2.3

Management of Warehouses

To have a functioning warehouse, it is important that the processes are well managed. Management of the warehouse can be done at three levels; strategic, tactical and operational level. At the strategic level long term decisions will be made as in what resources to invest. Tactical decisions are medium term decisions and concern the number of resources and the design of the layout. At the operational level short term decisions are made concerning the planning and control of the inventory, equipment and the assignment of tasks [23]. These decisions are made regarding the optimization of the processes.

2.3.1 Planning

The planning focuses mainly on the storage location and the processing of incom-ing orders. Van den Berg [26] introduces a four step location planning procedure containing the following steps:

2. Clustering of correlated products.

3. Balancing of workload within warehousing systems. 4. Assignment of products to storage locations.

The distribution of products among warehousing systems depends on their charac-teristics. In warehouses with a lot of variety within the products sometimes each warehousing system is equipped for a certain group of products. It also depends on the storage policy a warehouse has where items need to be stored. A dedicated storage policy means that the storage locations for each product are prescribed while a random policy leaves the decision to the order picker. In some case a class based policy is used. This policy assigns certain products to certain zones, but the loca-tion for storage in the zone can be randomly picked. If certain products are often ordered together a correlated storage policy or family grouping policy can be used [23]. This ensures shorter traveling times for the order pickers. Correlated products can be identified using historical data. The storage locations can change over time as popularity of products changes. To improve the capacity of the system Van den Berg [26] suggests to balance the workload within the warehousing system. Prod-ucts may be divided over the zones in such a way that mean and peak workload are evenly distributed over the order pickers assigned to each zone. A zone that barely receives orders and where order pickers are idle most of the time are not efficient.

With the handling of the incoming orders, the warehouse planning system also knows if the stock is still sufficient. It needs to constantly check if this is still the case. Whenever the stock tends to run low, new products from the suppliers have to be ordered.

2.3.2 Control

If an order is received, it will be released for picking. Control problems of a ware-house are: batching of orders, routing and sequencing and dwell point positioning [26]. In the problem of batching a trade-of has to be found between efficiency and ur-gency. It would be very efficient to only pick orders in very large batches at a nearby location, but this way orders with items located at the far end of the warehouse will end up being delayed. Therefore a rule of urgency has to be added to make sure all orders are handled as well. Also picking in very large batches will cause a large re-sponse time. To achieve this trade-of one can choose a static approach, in which the order picker works in blocks. First the optimal route in the most urgent block has to be found, before the second block with less urgent order picks is released. The other possibility is a dynamic approach. In the dynamic approach due dates are assigned to each order and a schedule needs to be established that satisfies the due dates. A lot of algorithms using heuristics have been made to minimize the travel time for retrieving the items. Gu [24] describes the problem as "determine a partition of orders for assignment to waves and pickers". The wave is the time window in which the order needs to be fulfilled. The optimization problem is subject to performance criteria and constraints as: picker effort, imbalance among pickers, time slots, picker capacity (based on volume, weight, etc.), and order due dates. Most of the solutions used for this problem are based on the Vehicle Routing Problem (VRP), in which one starts with one order. More orders will be added close to the route of the first

order, until the capacity constraint is reached. Another often used heuristic is the savings heuristic. First each order is assigned to one batch. After this the batches will be combined based on savings, until one of the constraints is reached.

Routing and sequencing determines the best route and sequence of items that need to be picked. Once all the orders in a batch are known, the routes between the lo-cations of the items need to be determined. The objective is to minimize the travel time between the pick up locations. Assumed is that the locations to be visited are given. For the different systems used for order picking, different solutions ex-ist. The solutions however are all based on the Traveling Salesman Problem (TSP) in which the shortest possible route to visit all locations will be calculated. For the picker-to-part system also heuristics based on traversal and return policies are used. The traversal policy states that the picker will always enter the aisle where a product needs to be picked at one side and leaves on the other. In other words, the order picker will travel through the whole aisle. The return policy states that the order picker enters and exits through the same side of the aisle. An aisle can be visited more often than just one time from the other side. An important addition is that the solutions should also consider congestion problems. In most warehouses the aisles are narrow and it is not always possible to pass each other, so two different order pickers can not be at the same location at the same time or can not cross each others in the same aisle [24].

When a AS/RS system is idle, the S/R machine stops moving. It will reside at a dwell point. The place of this dwell point is chooses in such way that the travel time to the first location after idle time is minimized. If the first assignment after idle time is a storage assignment, the optimal dwell point would be at the input/output station. One can however never be sure the first assignment is a storage assignment. Another way of solving this problem is to find a place that minimizes the maximum travel time [26].

Vis [25] mentions the following activities that need to be controlled in an AGV system:

• dispatching of loads to AGVs • route selection

• scheduling of AGVs

• dispatching of AGVs to parking locations

In some systems control can be done offline as all the orders are known in advance. In this case a centralized system is able to make all the decisions on dispatching, routing and scheduling in advance. However in warehouses this is not a common situation. As orders can come in during the whole day, the system must be capable of making real time decisions. In the system there should be a database available containing the pick up and delivery points.

Dispatching of loads to AGVs

becomes available that needs to be assigned to an AGV or when an AGV becomes idle and needs to be assigned to a task. The simplest solution to this problem is the first-come-first-served rule. Another possibility is the first-encountered-first-served rule. The AGV that first encounters the location of the load will be assigned with the task. These two rules can be used for AGVs traveling in a loop. Other heuristic rules that can be used are: random vehicle rule, in which the task is assigned to a random vehicle, nearest vehicle rule, in which the task will be assigned to the vehicle with the shortest distance to the pick up location, farthest vehicle rule, in which the vehicle farthest from the load receives the new assignment, longest idle vehicle rule, in which the AGV that remained idle for the longest time receives the task and least utilized vehicle rule, in which the task is assigned to the AGV with the lowest mean utilization. When looking at the second case (an AGV becomes idle and needs to be assigned to a task) the following rules for the dispatching problem can be used: random workcentre rule, in which the AGV chooses a random task, shortest travel times/distance rule, in which the AGV is dispatched to the task closest by, maxi-mum outgoing queue size rule, in which the AGV chooses a task in an area with the largest number of pending tasks, modified first-come-first-served rule, in which an AGV is dispatched to areas in chronological order to their time of request. These are all simpler heuristic rules. Other rules are based for example on Markov decision processes, fuzzy logic and neural network approaches. However, Vis [25] concludes that these hardly outperform the more traditional heuristics. The most important aspects are low computation times, deadlock avoidance and interaction with other equipment.

Routing and scheduling

After an AGV has been dispatched, a path to the pick up location and between the pick up and delivery location has to be made (route) and the related departure and arrival times have to be made (schedule). The schedule is especially important when the AGV system connects to a automated conveyor or packaging system. Also the departure and arrival time at each intersection is known, so collisions and deadlocks can be avoided. One of the objectives of the routing is to minimize travel time. Two different ways of routing can be distinguished: static routing and dynamic routing. In the first way all the distances between the nodes are known in advance. This way using different methods the shortest route can be easily calculated. The drawback of this way of routing however is that it does not take into account any changes in the system as traffic congestion. With dynamic routing it is possible to take congestion into account. For example by adding penalties to congested routes.

Dispatching of AGVs to parking locations

The same that goes for the dwell point in the robot for AS/RS also goes for the location of dwell points for AGVs. If an AGV becomes idle it has to move to a point so it does not cause traffic congestion, but at the same time is as efficient as possible when receiving a new task. In current systems often a new task can not be started before the AGV reaches its dwell position. Three different rules can be used for parking AGVs. The first is the central zone positioning rule, in which separate areas for parking are made in the warehouse layout. The second is circulatory loop positioning rule, in which AGVs keep traveling in the same loop, until they are dispatched. The third is the point of release positioning rule, in which an AGV

parks at the point where it is unloaded. This rule however can cause congestion at the unloading points. Important is that within large systems AGVs are able to be disturbed in their parking task and are able to start a new loading task, because of the great distances an AGV might have to travel to its dwell point.

2.3.3 Current Systems

Figure 2.10: The different layers when using ware-house management and wareware-house control systems. (Source: [2])

Nowadays many warehouses use warehouse control systems and warehouse management systems. A warehouse manage-ment system is mainly used to schedule shipments of incoming orders, manage inventory, stor-age locations and the workforce. Its focus is on the broader ac-tivities and it is able to fore-cast activities, using non real time data. It knows everything about the items in the ware-house as size, how it is packed and the storage locations in the warehouse. Whenever an order

comes in with items in random order, the WMS makes the pick list so the items can be easily picked. Gharbi [27] describes the basic function of a WMS as "pro-viding information to allow effective control of the movement of materials within the warehouse". Where WMSs are used for planning, WCSs are responsible for the execution. They can be seen as floor managers. WCSs manage the equipment on the floor and work with real time data. These systems often work separate from each other and find themselves in different layers of control, figure2.10. The imple-mentation of these systems however does not mean that a warehouse is automated. Often these systems are only used as support systems. Advanced systems do exist however. With the introduction of bar codes certain processes can be automated using bar code scanners. Also the usage of RFID chips ensures the the possibility to automate. Storage positions in the warehouse are easy to trace. As early as 1983 Karlstrom [28] describes techniques for automating a warehouse. He argues for a system that keeps track of the stock and control materials handling. The differ-ent functions of a warehouse that can be automated are the processing of incoming orders, the order picking itself, the stacking and packing of goods, the scheduling of trucks to pick up and deliver goods and the control of the stock. Figure 2.11

shows an example of the distribution of the different warehouse processes over a warehouse system. In this case the system is divided in two sub-systems. The first is a inventory planning and control system and the second is a a materials handling system.

2.4

Shortcomings

This traditional system for automating has a lot of drawbacks [9]. The first is that it is a costly system, because a lot of equipment has to be bought. Second is that

Figure 2.11: General model of a warehouse system. This system can be divided into two subsystems [10].

these systems take a long time to design and implement, because they have to be designed for a specific warehouse. Some are designed for a existing warehouse, so the layout is already determined. But even when newly designed, depending of the amount of different sorts of products, quantity of products and way of order picking every warehouse is different. This leads to the third drawback that these warehouses are inflexible. Once the equipment has been build and installed, it is hard to move them for they are very big and designed in such way that everything connects. Also changes in the inventory are hard to implement. For the same reason this makes it hard to expand the warehouses, which is the fourth drawback. The fifth is that most of the processes is done in batching. This causes the need of expensive sorting machines. Sixth, all products need to be stored in fixed locations, so the WMS knows where which product is as the storage locations are determined in advance. Finally, reslotting still needs to be done manually as products have to be moved from one storage location to another for efficiency reasons.Rouwenhorst [23] describes the downside of the software. Most of the WMSs are very complex in their design. This makes them hard to maintain and customize. The rules that guide the system are picked from a set beforehand. So the system always work by the same rules and does not optimize anything by learning.

2.5

Requirements

As discussed in this chapter, a warehouse is a complex dynamic system. A wide range of processes are happening at the same time and they all need to be con-trolled. There needs to be control of the planning of the inventory, but also the order picking system needs to be controlled. All the processes are distributed over different types of equipment. For this reason the different pieces of equipment need to be able to handle tasks autonomously. If the different components work inde-pendent from each other, it is important that there is communication between the equipment to optimize the handling of the orders. The system needs to be able to fulfill receiving tasks, put-away tasks, order picking tasks, scheduling tasks, manage

the inventory and storage control. It also has to be able to handle incoming orders and communicate with the customer. When one central system has control over the whole warehouse, all these processes will take a lot of computing time. Therefor a decentralized system is needed. All components in the different processes should be separated and divided into smaller systems. Because a lot of different warehouses and equipment exists the generality of the system means that is more easily applied in different warehouses. Also the system needs to be controlled real time as the environment is very dynamic.

3

Intelligent Agents

Before giving an overview of the implementation of agent technology in warehousing, it is important to have an understanding of the concept of agents and agent systems. Agent technology can be grouped in three categories: organization level, interaction level and agent level. This chapter will focus on the agent and interaction level and describe what an agent exactly is and how it is build and explains why agent technology is highly suitable for warehousing systems. The second part of this chapter explains how agents work together and can form multi agent systems.

3.1

What is an Agent?

As we consider agents we mean intelligent software agents that are used for agent systems. Software agents originate from AI research. Basically an agent is a way of designing software. This can be done in different programming languages as C++ or Java. Object-oriented software can be used to design agents. Whereas an object describes the classes, methods and attributes of a software component, an agent is defined by its behaviour [29]. The design of agent software however lies outside the scope of this report and will not be discussed. There are different sorts of intelligent software programs, so a good definition of an agent is necessary. The most complete definition is the following: An agent can be described as an autonomous computer system, situated in an environment which it can perceive and on which it can act in order to meet its design objectives [11] [30] [31] [32]. So basically an agent is a software program that can be programmed to reach certain objectives. It receives its information about the environment through sensors (perception). Based on this information the agent then computes the best actions to reach its goals and finally executes these actions.

A schematic drawing of an agent in general is shown in figure3.1. As can be seen the agent receives information through sensors from the environment and can perform actions in the environment using actuators. Based on how the agent is programmed and the data it already possesses, it makes decisions on which actions to take in the environment.

Figure 3.1: Schematic drawing of an intelligent agent. It receives percepts from its envi-ronment by sensors and can perform actions on the envienvi-ronment with the help of effectors (actuators). Source: [11]

Figure 3.2: Examples of PAGE [11]. The different percepts, actions and goals are given for different agent types.

Agent technology has been introduced in computer networks, software engineer-ing, artificial intelligence, human-computer interaction, distributed and concurrent systems, mobile systems, telematics, computer-supported cooperative work, control systems, decision support, information retrieval and management, and electronic commerce [33]. An agent can be as simple as a square root function on a calculator, where perception is the press of the keys and action is displaying the correct answer to complicated systems such as an air traffic control system. There are many differ-ent types of agdiffer-ents. Agdiffer-ents can be fully in control of the actions they perform in the environment they are situated in or they can support humans in making decisions. The systems in warehousing are mainly control systems. They control the different processes in the warehouses and the equipment used in these processes. Russel [11] states that before designing an agent, one should know what the perceptions, the actions, the goals of the agent are and in what environment the agent is situated (Percepts, Actions, Goals, Environment, PAGE). Figure 3.2 shows some examples of different types of systems that involve agents and their corresponding PAGE.

Percepts: The percepts can be received by sensors. There are many different sensors. Examples of parameters that can be measured using sensors are: position, motion, velocity, displacement, acceleration/tilt, flow, sound, vibration and temperature. Often used sensors to perceive these parameters are (3D) cameras, infrared cam-eras, accelerometer, gyroscope, microphone, thermometer, GPS, sonar and RFID sensors. These are all sensors that can measure the physical environment and can be used for example for order picking systems. Especially AGVs need to be able to

avoid obstacles and are usually equipped with these sensors. Percepts can also be software based. For planning and stock control it is the information in a database about the inventory and the incoming information through scanners. Another way to achieve information from the environment comes from input by humans.

Actions: The actions the agents can take can be in space (e.g. movement by chang-ing speed and, rotation position or apply a certain force) as well as software based (e.g. communicate, send information to other agents, display messages). The re-sources an agent has to perform actions can be CPU and memory in a pure com-putational setting, while in a physical system they can be elements in space as equipment. Again here the distinction between the order picking system, operating in the physical environment and the planning and scheduling system, both operating in a software based environment can be made. The actions an AGV for example can take are accelerate, decelerate, turn, load and unload, while the actions of a planning and scheduling system can be send pick list to AGV or send a reply to the customer.

Goals: The goals, also called the objectives, of the simplest agents are no more than reacting on a certain situation in a certain way (if, then). The goals of most agents however can be defined by minimizing or maximizing certain parameters as costs, time or distance. Other often used goals are establishing a certain state of the environment. For an order picking system the goals can be to minimize the picking time or to minimize the travel distance.

Environment : The environment, an agent is situated in, can be anything from a software environment to a physical environment where an agent has to move past obstacles. In a warehouse the environment can be a software environment as well as a physical environment. The planning and control of the warehouse system is mostly done in a software environment, while the order picking and the other warehouse processes are done in the physical warehouse itself.

Besides control in warehouses, agents can also be used for simulation. Complex and dynamic environments that are hard to model can now be easily modeled using agents. These models can be used to provide answers to complex real-world prob-lems, involving a lot of different components and can provide a tool for management systems [33]. In warehouses for example simulation can be used to find the optimal lay-out of the warehouse, optimizing the travel time of the order pickers and the placement of products based on popularity and the optimal storage location of the stock. Also replenishment tactics can be found using models, should the stock be stored in the same aisle or in a completely different part of the warehouse? These questions can be answered using models.

3.2

Characteristics

The environments agents are situated in are constantly changing. This means that to reach its goals, an agent must rapidly react on its environment. To fully function, an agent has to possess certain characteristics. Agents can be described using mul-tiple characteristics. Some of them are already mentioned in the definition above. Wooldridge [34] attributes agents with the following characteristics:

• Autonomous • Reactive • Proactive • Social

Autonomous means the system should be able to function without the intervention of humans, so continuous supervision is not needed. It can make decisions over its own actions and control its internal state and does not need a centralized computer to control the agent. It is in control over how it reaches its goals. In other words, it is independent. As discussed in the previous chapter, this is one of the requirements of a warehouse system. Because there are so many components in such a system, it would take too much computing time to control all the components with a cen-tralized system. Therefor the different components and processes in warehouses can be divided over different agents. Every order picker for example could be seen as a separate agent, or there can be an agent operating as a planning agent, keeping track of the inventory. These two agents do not have anything to do with each other and can work on their own tasks, without the intervention of a human or a central computer. They act as human operators who have responsibility, show initiative and are autonomous.

Reactive means that an agent is able to react to changes (which they perceive) in the environment they are in. The agent must be prepared to adapt its plans to the changes in the environment. For example an AGV driving to the pick location the item is stored. In the warehouse environment more than just one AGV is active. So a certain AGV has to be able to react to the movements of the other AGVs in the warehouse.

Proactive agents do not only react to changes in their environments, but they also have goals and take actions to reach those goals. An AGV agent can have a goal of picking up an order in a as short as possible time. At the same time the goal of the agent can be to take the shortest route to the pick location. The goal of an agent concerned with the planning of tasks can have a goal of assigning the AGV to items lying as close as possible to each other or make a picking list with as many possible items in a certain time span. In agent technology other terms for goal that can be used are: task, objective, aim and desire.

Social agents are able to work together with other agents and communicate so they are also able to reach an overall goal, besides their own goals. This is very impor-tant considering the fact that agents are autonomous. In some cases agents have to work together to reach overall goals. In this case agents need to communicate. The communication between agents will be discussed in section3.5.

It is important that an agent finds a balance between being reactive and being proac-tive. The actions of an agent are influenced by its goals as well as its environment. If an agent is too reactive, then it will constantly change its actions according to its environment and will not achieve its goals. If an agent is too proactive, it will try to reach its goal, but will not react to its environment enough, so the actions it

performs are no longer relevant.

Next to these four characteristics, Padgham [35] distinguishes four more agent char-acteristics. He states that agents are also:

• Situated • Flexible • Robust • Rational

Situated means that the agents are situated in an environment that is dynamic, unpredictable and unreliable. Being situated in a dynamic environment means that while trying to reach a goal the environment does not stay the same. For example in a physical environment objects can move around, so the location of AGVs in a warehouse changes all the time. In a unpredictable environment it is not possible to predict the future states of the environment. This also happens in a warehouse. It is not possible to predict the incoming orders and therefore the assignment of tasks to other agents. An environment is unreliable whenever the actions an agent takes may fail. Especially in the physical environment this can be the case when the path of an AGV is blocked and it can not move that way anymore or when a load has been dropped.

Flexible. In case the action the agent takes fails, it is important that agents are flexible. This means that the agents are able to find an alternative for reaching their goals.

Robust means that an agent is able to recover from failure. For example if the AGV gets stuck, because of a blocked path, it will start over and rethink a way of reaching its goal. Flexibility and robustness are closely related as robustness can be achieved by having a flexible agent.

Rational means that the agent does not simultaneously performs two conflicting actions. When ordering new supply, one can not spend the same money on two products. A rational agent can consider both actions, but finally chooses the one that is the best solution for reaching their goals. An overview of all agent characteristics are shown in table3.1.

Table 3.1: Agent Characteristics Characteristic Description

Autonomous The ability to function without intervention of humans. Reactive The ability to react to changes.

Proactive The ability to take actions to reach goals.

Social The ability to communicate with other agents to reach overall goals. Situated Agents are situated in an environment that is dynamic,

unpre-dictable and unreliable.

Flexible The ability to find an alternative to reach the goal. Robust the ability to recover from failure.

3.3

Classes of Agents

In single-agent systems, there is only one agent controlling the system. This agent has access to all actuators and sensors and has full control over the system. Russel [11] distinguishes four different kinds of agents based on their intelligence, table3.2.

Table 3.2: Different sorts of agents

Agent Sort Description

Simple reflex agent decision making based on if/then rule

Reflex agent with internal state makes decisions based on changes in environment Goal-based agent actions based on evironment as well as goals it wants to

achieve

Utility-based agent try to reach goals in the best possible way

Simple reflex agents are the simplest kind of agents and are build based on the if-then rule. The agent perceives the environment and takes actions according to the pre-programmed rules. For example in automated driving. The pre-programmed rules can tell the agents for example to brake if the breaking lights of the car in front light up. If the lights light up, then break.

Reflex agents with internal state store former perceptions of the environment in its internal state. This makes it possible to compare the current state of the en-vironment to see how it evolves and make decisions based on the changes in the environment. In case of the breaking lights on a car, the agent can compare former perceptions of the car in front and compare these with the incoming images. The breaking lights have to be in exactly the same spot for a simple reflex agent, because otherwise it might recognize the regular back lights of the car as breaking lights. To distinguish the breaking lights from the lights normal lights of the car it is useful to compare images as breaking lights only light up for a short time.

Goal-based agents are not only aware of their environment, they also have goals they want to achieve in that environment. Their actions are therefore based on the perceptions as well as on the goal the agent has to achieve. The goal based agent in the car example might also be aware of the breaking efficiency. If the goal is not hitting the car in front of you, the timing of breaking also comes into play. If the roads are wet, the car might need to break earlier than it has to do in dry weather. So with adding a goal, the agent becomes more intelligent and can make decisions not only based on incoming information and its internal state, but also based on future situations.

Utility-based agents not only try to reach their goals, but do it in the best possible way. They will always try to find the most optimal solution at the moment (e.g. cheaper, safer, or faster). For example in routing of a vehicle. There are many possible routes that lead to the destination, but not every route is as efficient. A certain route may be shorter and safer than another and therefor be a better option than other possible routes.

As has been shown above, there can be many different levels of intelligence pro-grammed in an agent. They all meet the definition of an agent; an autonomous computer system, situated in an environment which it can perceive and on which it can act on in order to meet its design objectives, however most complete and intelligent agents are utility based agents.

3.4

Decision Making

These different sorts of agents all make decisions in different ways. This section will zoom in more on the decision making structures within agents. Woolridge [30] divides four different agents when it comes to decision making:

• logic based agents • reactive agents

• belief-desire-intention (BDI) agents • layered agents

The first two types of agents named by Woolridge are fairly simple agents. The logic based agents make decisions using symbolic reasoning. The environment is repre-sented by symbolic formulas. For example: If situation P is occurring, then R. The agent can reason for itself the state of the environment and which actions to perform.

Reactive agents don’t maintain an internal state. They make decisions based on rules and only take actions in response to changes in the environment.

Belief-desire-intention agents were first introduced by Bratman [36] in 1987. Be-cause an agent can not always be aware of everything happening in its environment some agents have beliefs. This means that if something happened before, it gets stored in the internal state of an agent, the beliefs of an agent. The agent updates its beliefs constantly by the information it receives through sensors. The desires are the goals the agent has to achieve and the intention are the chosen options upon which the agent will act [35]. Many researchers have been developing agents based on this decision method.

In layered architectures the agent consists of different subsystems that all make decisions. Two different layered architectures can be distinguished: a horizontal architecture and a vertical architecture, figure3.3.

Figure 3.3: Layered architecture of an agent (Source: [12])

All the layers in the agent have access to all of the perception and action com-ponents [12]. In a horizontally layered architectures before a decision is made the information first have to pass all layers concurrently. It can consist of a control layer, reactive layer and planning layer. To control what passes through what layer, there are several global control rules. This is also the downside of horizontal layered architectures. Therefore more research has been done on vertical layered architec-tures. In this architecture control flows from the lower levels to the upper levels, so the lower layers are crucial. In this architecture no centralized control rules are needed.

3.5

Agent Systems

The ability to design intelligent agents leads to a computing that is social. This section will focus on the interaction level of agent technology.

Logistic systems are often large and dynamic as there are a lot of actors involved [37] [38]. While it is possible for certain systems to be handled with only one agent, in the field of logistics most intelligent systems consist of multiple agents, because of the complexity of logistic systems. In these systems the agents work together not only to reach their individual goals, but also reach an overall goal. These systems are also called Multi-agent systems (MAS). Sycara [39] assigns the following characteristics to MASs:

• not every agent has all the information of the system and therefor has a limited "viewpoint"

• the system is not globally controlled • data are decentralized

So it can be said that there is not one agent that possess all the information. A multi-agent system consists of different agents that can interact with each other. For MASs there are different types of systems. The main difference can be found in the structure of the systems.

3.5.1 Structure

The structure of a MAS can be seen as the roles of the agents the MAS consists of and how they communicate with each other [40]. Davidsson distinguishes two types of structures; a structure is either static or changes over time and is therefore dynamic (also called an open system by Sycara [39]). The structure of a system can be described by the different agents involved, their roles and the communication between agents.

• A static structure means that the agents do not change during execution of a task. There are always the same agents involved in the task.

• A dynamic structure changes over time. This means that the agents as well as the environment can change over time.

Almost all logistic agent structures are dynamic, because the transport part of the chain takes place in a constant changing environment if one takes into account the dynamic changes of transport demands, intermodal freight transport network prop-erties and traffic conditions in the network. Also the agents involved in the system can change over time. For example at a container terminal. Different vessels (agents) come in only to leave when the loading or unloading has been finished. The same goes for warehousing. If a warehouse consists of multiple AGVs (for order picking) the tasks that come in are never assigned to just one and the same AGV. Instead the tasks will be distributed over the AGVs depending on their dwell position and the distance to the pick up location. This also means that the environment is constantly changing as the location of the AGVs does not stay the same, but varies over time.

Different agents in such systems can have different tasks. Each agent focuses on its own task and tries to reach its own goals. Fox [41] lists the following issues when designing a system:

• What activities to assign to what agent? • How do the agents interact?

• How quickly does an agent respond to its environment?

In the case of warehousing most of the processes can be divided over different agents as will be discussed in the next chapter. Fox states that the planning and scheduling function is the most important function and "orchestrates" the behaviour of the other agents, but the most important thing is that the system is complete and that the total functionality of the agents covers all the processes and functions of the warehouse. The second question depends on the different ways agents are able to respond and how they cooperate in achieving a solution. Fox states that they should interact in such a way that it can cooperate with other agents in finding a solution.