Sampling FEE and trigger-less DAQ for the J-PET scanner

SAMPLING FEE AND TRIGGER-LESS DAQ FOR THE J-PET SCANNER

G. Korcyl^a, D. Alfs^a, T. Bednarski^a, P. Białas^a, E. Czerwiński^a K. Dulski^a, A. Gajos^a, B. Głowacz^b, B. Jasińska^b, D. Kamińska^a Ł. Kapłon^a,d, P. Kowalski^f, T. Kozik^a, W. Krzemień^c, E. Kubicz^a

M. Mohammed^a, Sz. Niedźwiecki^a, M. Pałka^a

M. Pawlik-Niedźwiecka^a, L. Raczyński^f, Z. Rudy^a, O. Rundel^a N.G. Sharma^a, M. Silarski^a, A. Słomski^a, K. Stoła^a

A. Strzelecki^a, A. Wieczorek^a,d, W. Wiślicki^f B.K. Zgardzińska^b, M. Zieliński^a, P. Moskal^a

aFaculty of Physics, Astronomy and Applied Computer Science Jagiellonian University, 30-348 Kraków, Poland

bDepartment of Nuclear Methods, Institute of Physics Maria Curie-Skłodowska University, 20-031 Lublin, Poland

cHigh Energy Physics Division, National Center for Nuclear Research 05-400 Otwock-Świerk, Poland

dInstitute of Metallurgy and Materials Science of Polish Academy of Sciences 30-059 Kraków, Poland

eFaculty of Chemistry, Jagiellonian University, 30-060 Kraków, Poland

fŚwierk Computing Center, National Center for Nuclear Research 05-400 Otwock-Świerk, Poland

(Received January 18, 2016)

In this paper, we present a complete Data Acquisition System (DAQ) together with the readout mechanisms for the J-PET tomography scanner.

In general, detector readout chain is constructed out of Front-End Elec- tronics (FEE) measurement devices such as Time-to-Digital or Analog-to- Digital Converters (TDCs or ADCs), data collectors and storage. We have developed a system capable for maintaining continuous readout of digitized data without preliminary selection. Such operation mode results in up to 8 Gbps data stream, therefore, it is required to introduce a dedicated mod- ule for on-line event building and feature extraction. The Central Controller Module, equipped with Xilinx Zynq SoC and 16 optical transceivers, serves as such true real time computing facility. Our solution for the continuous data recording (trigger-less) is a novel approach in such detector systems and assures that most of the information is preserved on the storage for fur- ther, high-level processing. Signal discrimination applies a unique method of using LVDS buffers located in the FPGA fabric.

DOI:10.5506/APhysPolB.47.491

(491)

1. Introduction

The prototype of TOF-PET scanner, constructed by the J-PET Col- laboration [1–11], consists of 192 plastic scintillators, each equipped with photomultipliers (PMTs) on both ends. This results in 384 analog chan- nels that have to be processed by the Data Acquisition System (DAQ). The J-PET prototype will be used for investigations in the field of medical imag- ing [5, 12], nano-biology [13], material science [14, 15] and for testing of fundamental symmetries in physics [16, 17]. In this paper, we present a complete solution for the DAQ system together with the readout mecha- nisms. Detector readout chain is constructed out of Front-End Electronics (FEE), measurement devices such as Time-to-Digital or Analog-to-Digital Converters (TDCs or ADCs), data collectors and storage. Most of the PET scanners include coincidence units or multi-level trigger logic in order to dis- card in the real-time data classified as background noise [18, 19]. Such a unit can only execute low-level selection algorithms in order to fulfil real- time regime. Applying low-level rejection filters can result in the loss of fraction of valuable data. On the other hand, more complex algorithms might introduce longer dead-time of the DAQ system, resulting in reduced rate of registered events. Our solution for the continuous data recording (trigger-less) is a novel approach in such detector systems and assures that most of the information is preserved on the storage for further, high-level processing. The core of the presented system is based on Trigger Read- out Board v3 (TRBv3) platform [20,21], developed for and widely used in high energy physics experiments [22].

2. System overview

The main element of the J-PET DAQ system is the collection of TRBv3 modules. Those are high-performance and FPGA-based (Field Programma- ble Gate Array), therefore reconfigurable, TDC readout boards. Each mod- ule is equipped with five Lattice ECP3M devices. The central one serves as the controller and local data collector, while the remaining four can be con- figured with configware providing various functionality. For the precise time measurement, a design providing 48 input channels and time resolution of 12 ps has been developed [23]. Each input channel has rising and falling sig- nal edge detection and a buffer for up to 54 complete signals storage between two consecutive readouts. One TRBv3 module is called Master and controls the readout procedure and synchronization of all the other modules (Slaves), Fig. 1. Each TRBv3 board has an individual Gigabit Ethernet (GbE) link for transmitting collected measurement data out of the system for storage with the use of standard and cheap network facility. An additional module, called Central Controller Module (CCM), has been developed in order to

provide efficient, on-line data processing. The module has sixteen GbE links as inputs from the Slave modules and a Xilinx Zynq-7045 as the processing unit. Data packages sent from the Slaves can be directed through the CCM, which can perform on-line analysis, histogramming and data quality assess- ment. The data is saved on the Event Building (EB) machines. Those are server-class, multiprocessor PCs running software for collecting data frag- ments from the network and reassembling them into complete data units called Events, representing the state of the entire detector in a particular period of time. Such files are taken as the input into the analysis and image reconstruction algorithms [24,25].

Fig. 1. Schematic view of the JPET Data Acquisition Setup. The readout process is controlled by the Master TRBv3 module, whereas Slave TRBv3 modules perform digitization of detector signals coming through Front-End Modules. The readout data is streamed to the Central Controller Module and then further to permanent storage.

2.1. Front-End Electronics

Signal shaping, amplification and discrimination are the main functions of the Front-End Electronics [26]. A dedicated electronic module has been developed as the mezzanine Add-on for the TRBv3 boards. The module is plugged directly into the connector which provides power, data and control lines. The signal discrimination applies a novel concept of misusing (using in a non-standard way) the LVDS buffer inside the FPGA device, which is configured with time measurement logic. An LVDS buffer has two inputs:

positive and negative. The output state of the buffer changes the logic state in case the voltage levels on the inputs are crossed. One can apply on one of the inputs a threshold level, and the analog signal on the second input. At the moment when the analog signal crosses the threshold level, the buffer will generate a logic pulse, which being already inside the FPGA, can be directly inserted into the tapped-delay chain for precise time measurement.

The input signals are amplified and split into four paths, each having an individual threshold level. It is a realization of the multi-level threshold- ing concept as a measure for reducing the time-walk effect and, therefore, achieving better timing resolution [27]. The TDC design allows for measure- ment of 48 input channels, hence the FEE module has 12 inputs from the photomultipliers.

2.2. Readout procedure

The setup for the J-PET prototype consists of eight TRBv3 boards as Slave modules and one Master TRBv3. The readout process in the system is controlled by the Master module, which has the Central Trigger System (CTS) functionality implemented in the central FPGA device. For the con- tinuous type of readout [28], the CTS sends a periodic Readout Request message to all the Slave modules at the fixed rate of 50 kHz. The Slaves record input signals and store the data in buffers until Readout Request message arrives. At the moment the message arrives, the buffers are cleared and the data is encapsulated into UDP packets and sent over Gigabit Ether- net network to the Event Building machines. A single UDP packet contains measured time samples recorded by one Slave modules during a period of 20 µs. The readout rate and buffers sizes (up to 54 signals per channel) are adjusted accordingly in order not to get overflowed and, therefore, to achieve maximum signals acceptance. Upon receiving a Readout Request message, each module constructs a UDP packet, tagged with the Readout Request message sequence number and module ID. Those two values are necessary for the Event Building machines to correctly combine packets from the same measurement but different sources into one, single data unit.

2.3. Central Controller Module

The described readout type results in a significant constant stream of data that has to be processed. The maximum throughput that can be achieved reaches 8 Gbps. This amount of data can be efficiently distributed for storage over a number of Event Building machines running in parallel and connected with 10 Gb Ethernet network. However, the high event rate and the distributed data storage makes the designed system not suitable for on-line analysis. To overcome this drawback, an additional module called Central Controller Module (CCM) has been designed and developed. The Central Controller Module provides a computing facility for on-line analysis and data quality assessment. The board features a Xilinx Zynq-7045 FPGA device, which is a hybrid of FPGA resources and an ARM processor. The architecture of FPGA devices allows for natural parallelism for processing multiple data streams, while the standard processor provides a convenient access to the results. The module can be used as the “board-in-the-middle”, meaning it can receive the packet streams from Slave modules, perform anal- ysis and forward the original packets further, to Event Building machines.

TRBv3 data format parsers and feature extraction algorithms have been implemented as the foundation for higher-level data analysis.

3. Summary and outlook

A solution for the complete Data Acquisition System for J-PET proto- type has been developed and is under evaluation. It consists of two novel approaches: in-FPGA signal discrimination and continuous type of the read- out [29]. Those two techniques allow for collection of timing data, measured with the high resolution. At present, lack of real-time data selection and filtering mechanisms in favour of trigger-less readout results in significant amount of data for processing but reduces the risk of discarding valuable measurements. Ongoing work focuses on hardware evaluation and develop- ment of the on-line algorithms for the Central Controller Module.

We acknowledge technical and administrative support by A. Heczko, M. Kajetanowicz, W. Migdał, and the financial support by the Polish Na- tional Center for Development and Research through grant No. INNOTECH- K1/IN1/64/159174/NCBR/12, the Foundation for Polish Science through MPD program and the EU MSHE Grant No. POIG.02.03.00-161 00-013/09, the Polish National Centre for Research and Development through LIDER grant 274/L-6/14/NCBR/2015, Marian Smoluchowski Kraków Research Con- sortium “Matter–Energy–Future” and the Polish Ministry of Science and Higher Education through grant 7150/E-338/M/2015.

REFERENCES

[1] P. Moskal, Patent numbers: WO2011008119-A2; PL388555-A1;

US2012112079-A1; EP2454612-A2; JP2012533734-W.

[2] P. Moskal, Patent numbers: WO2011008118-A2; PL388556-A1;

US2012175523-A1; EP2454611-A2; JP2012533733-W.

[3] P. Moskal et al.,Nucl. Instrum. Methods A 764, 317 (2014).

[4] L. Raczyński et al.,Nucl. Instrum. Methods A 764, 186 (2014).

[5] P. Moskal et al.,Nucl. Instrum. Methods A 775, 54 (2015).

[6] P. Moskal et al.,Phys. Med. Biol. 61, 2025 (2016).

[7] L. Raczyński et al.,Nucl. Instrum. Methods A 786, 105 (2015).

[8] P. Kowalski et al.,Acta Phys. Pol. A 127, 1500 (2015).

[9] W. Krzemień et al.,Acta Phys. Pol. A 127, 1491 (2015).

[10] P. Moskal et al.,Acta Phys. Pol. A 127, 1495 (2015).

[11] A. Wieczorek et al., Acta Phys. Pol. A 127, 1487 (2015).

[12] P. Moskal et al., Patent number: WO2015028604-A1.

[13] E. Kubicz et al.,Nukleonika 60, 749 (2015).

[14] A. Wieczorek et al., Acta Phys. Pol. A 127, 1487 (2015).

[15] A. Wieczorek et al., Nukleonika 60, 777 (2015).

[16] P. Moskal et al.,Acta Phys. Pol. B 47, 509 (2016), this issue.

[17] D. Kamińska et al.,Nukleonika 60, 729 (2015).

[18] T. Frach, T. Solf, A. Thon, US Patent, US 8164063 B2.

[19] D.J. Schlyer, P. O’Connor, C. Woody, S.S. Junnarkar, V. Radeka, P. Vaska, J-F. Pratte, US Patent, US 7091489 B2.

[20] M. Traxler et al.,JINST 6, C12004 (2011).

[21] A. Neiser et al., JINST 8, C12043 (2013).

[22] J. Michel et al.,JINST 6, C12056 (2011).

[23] C. Ugur et al.,JINST 7, C02004 (2012).

[24] W. Krzemień et al.,Bio-Algorithms and Med-Systems 10, 33 (2014).

[25] W. Krzemień et al.,Nukleonika 60, 745 (2015).

[26] M. Pałka, P. Moskal, Patent number: WO2015028600-A1.

[27] M. Pałka et al., Bio-Algorithms and Med-Systems 10, 41 (2014).

[28] G. Korcyl, P. Moskal, M. Kajetanowicz, M. Pałka, Patent number:

WO2015028594-A1.

[29] G. Korcyl et al.,Bio-Algorithms and Med-Systems 10, 37 (2014).