A meta-analysis of the results of evaluation studies concerning the evaluation of support provided as part of the ESF2

An unquestionable advantage of using ZUS data is its full coverage for all support participants.

Thus, the problem of sample randomness (and the declarative nature of information obtained by means of surveys) when calculating the values of long-term result indicators was eliminated. However, the use of ZUS data to calculate indicators also involves certain limitations. In some forms of employment, social security contributions are not always paid. This applies to persons working on the basis of contracts for a specific task, students under 26 years of age employed under contracts of mandate, persons working in the grey economy, and those who run a farm and are insured in the Agricultural Social Insurance

21 Cf. Badanie efektów wsparcia…, 2018. It was a long-term study of the effects of support provided to young people (PO KED, Youth Employment Initiative – YEI). The persons covered by the support (study) are NEET (not in employment, education or training), and are 15–29 years old. The aim of the study was, among others, to calculate the value of four long-term result indicators (from the Common List of Key Indicators), measuring the effects of support provided to young people as part of OP KED, six months after the end of treatment. Cf. www.power.gov.pl/media/56886/III_raport_wskaznikowy_27_03_18.pdf [accessed: 15 January 2020].

22 Metaanaliza wyników badań ewaluacyjnych… 2018, pp. 207–208. This was a long-term study aimed at synthesising the results of the 2017 MA and IB evaluation reports on programmes using support from the ESF (OP KED, ROP) and estimating the value of selected indicators (from the WLWK list).

The persons covered by the support (study) are the participants of OP KED who are inactive in the labour market (not in employment, education or training), belonging to different social groups, six months after receiving ESF support. Cf. www.power.gov.pl/media/60892/Metaanaliza_EFS_RCz_2018.pdf [accessed: 15 September 2019].

97 Evaluation of support for competencies of people on the labour market and for employment promotion...

Fund (KRUS)23. Also, the analysis of earnings using ZUS data is limited. Apart from situations where ZUS contributions are not paid, contributions paid by the self-employed which are not income-related are also problematic24.

It is of note, however, that ZUS data indicating the survival rate of start-ups that received support (i.e. continuation of business activity by persons who started self-employment), together with

parameters concerning the situation of their personnel (the amount of contributions paid, approximate size of the remuneration fund) remain unavailable from other data sources used so far in evaluation studies (GUS statistical research, REGON register, KRS).

On the basis of the report entitled Metaanaliza wyników badań ewaluacyjnych dotyczących oceny wsparcia w ramach EFS [Meta-analysis of the results of evaluation studies on the evaluation of support under the ESF] and documenting the multi-annual (2016–2018) counterfactual analyses of ESF impact carried out on the basis of ZUS microdata, we can tell how many people were still running their newly started business 30 months after project completion (53%) and what the financial effectiveness of the support was (cf. Metaanaliza wyników badań ewaluacyjnych… Raport cząstkowy…, 2018, pp. 3–4)25. It is worth noting, however, that in the final report on the study, the micro approach was complemented by a mezzo perspective on the impact of the ESF on the local (county-level) labour markets. It tells us that:

1. The support provided to the unemployed has contributed to reducing registered unemployment at the local level;

2. The award of one thousand start-up grants in the years 2015–2017 contributed to an increase

in the number of natural persons conducting business activity in 2018 in the county of residence of project participants (on average by 760);

3. Reducing the number of the unemployed in the county by one person required eligible expenditure on support for registered unemployed persons amounting to PLN 7,000 on average (Tarsa, 2019, p. 16)²⁶. As the authors of the report indicate, the weakness of the counterfactual approach so far is that: it ignores the indirect effects on non-participants, especially the substitution effect. For example, thanks

to the support, a participant could get a job at the expense of a non-participant, which means a neutral impact of support on the employment level, but is interpreted as a positive net effect in the form of improvement of the participant’s professional situation. Likewise, setting up a new business with the help of ESF funds can contribute to driving existing businesses from the market. Therefore, the results of typical analyses help to assess the impact of support on the future fate of a participant, but do not provide information on the impact of the intervention on the labour market, including e.g. on the level of unemployment [at the local level – author’s note]. The approach used in this report helps to fill the knowledge gaps, as the impact of ESF interventions on [...] the number of registered unemployed people

23 The scale of underestimating employment success rate calculated on the basis of ZUS data is probably small [...] and reaches between 0.5 [...] and 3%.

(cf. ibidem).

24 Of those working six months after the end of their participation in support, 5% worked under more than one contract at the same time. In the case of persons insured under several different titles, the majority were concurrent employment contracts and contracts of mandate, which may result in the contract of mandate not requiring social security contributions (Badanie efektów wsparcia…, 2018, pp. 30–31).

25 Cf. www.power.gov.pl/media/60892/Metaanaliza_EFS_RCz_2018.pdf [accessed: 15 September 2019].

26 Cf. www.ewaluacja.gov.pl/media/79605/Metaanaliza_wynikow_badań_ewaluacyjnych_dotyczacych_ oceny_wsparcia_z_EFS.pdf [accessed: 15 September 2019].

in a participant’s county of residence is analysed. The results, therefore, take into account substitution and other indirect effects [...]²⁷.

The evolution of the approach noted in the last edition of this study – i.e. the combination of participant and local market perspectives – should therefore be assessed positively. Nevertheless, it can be treated as an intermediate stage between the evaluation from the microeconomic perspective and evaluation of ESF impact on the labour market in Poland and employment indicators at a macro level.

Counterfactual evaluation of programmes’ impact on the labour market (macroeconomic perspective)

As indicated in the introduction, the evaluation of effects from a microeconomic perspective is valuable, as it allows for checking whether an intervention mechanism is at all effective for participants and their individual situation on the labour market. However, if evaluation units and evaluators stop at the micro perspective, giving up the observation of the target area from “a bird’s eye view” (macro perspective), they accept lack of knowledge about indirect impact of the support on the wider programme environment (including on the market of service providers and other labour market actors). They will also have no knowledge of possible flows of effects and social change expressed in the programme objectives. This means that such evaluations in a sense reduce the social utility of the programme to only one perspective and provide an incomplete picture of the effects.

Analyses using macroeconometric modelling have a long history in economic research. Their role has also been highlighted in the area of monitoring and evaluation of the use of European funds.

Starting from the years immediately preceding Poland’s accession to the EU, various types of research works based on macromodelling and concerning ex ante estimated impact of EU funds on the level of basic macroeconomic indicators of the Polish economy (GDP, employment/unemployment,

investments, etc.) were systematically commissioned by ministers responsible for regional development.

The latest such analysis was commissioned in 2019 by the Development Strategy Department of the Ministry of Investment and Development. It concerns the impact of cohesion policy on the social and economic development of Poland and its regions in the years 2004–201728. It also takes into account estimates of the expected impact of funds awarded as part of the current perspective for 2014–2020, both at the national and regional level. Moreover, the analyses document historical impact of the funds since 2004 and provide forecasts until 2023 concerning, among others, employment and income levels of the residents. For example:

In 2018, the employment rate of people aged 20–64 in Poland was 72.2% [...], and if not for EU funds, the employment rate [...] would have been 70.3%. [...] The strongest impact was recorded

in the Warmińsko-Mazurskie Province (3.0%) and the Podkarpackie Province (2.6%) [...];

27 This approach also has certain limitations. It ignores effects occurring outside a beneficiary’s county of residence [...]. None of the approaches described, neither these focussed on the participant’s future nor these concentrating on the local labour market, are ideal, as they do not present the full picture.

They are, however, complementary and, combined, allow for a much more complete and multi-faceted evaluation of the intervention (Metaanaliza wyników badań ewaluacyjnych… Raport cząstkowy…, 2018, pp. 253–254).

28 Cf. www.ewaluacja.gov.pl/media/79609/Wplyw_polityki_spojnosci_na_rozwoj_spoleczno-gospodarczy_Polski_i_regionow_ w_latach_2004-2018.pdf [accessed: 15 September 2019].

99 Evaluation of support for competencies of people on the labour market and for employment promotion...

1. When Poland joined the EU, it had the highest unemployment rate (19.1%). In subsequent years, it fell rapidly to 3.8% in 2018. At that point, it was 3% lower than the EU-28 average [...]; and 1%

lower due to EU funds. In absolute terms, this means a decrease of the number of the unemployed by more than 400,000 [...];

2. Compared to 2004, a per capita income in comparison to the EU average increased by 19.3%²⁹. Initially, such works were connected with relatively detailed requirements of the European

Commission (EC) and concerned extensive assistance measures (entire perspectives of the NDP/CSF 2004–2006, NCS/NSRF 2007–2013). In recent years (2015–2019), macroeconometric literature – in particular commissioned by PARP, the Educational Research Institute or selected ministry

departments, which act in the capacity of Operational Programme Managing Authorities (SG OP, EP OP) as an element of evaluation of a given programme – is also starting to focus on selected target areas of fund utilisation. These areas are e.g. social and economic cohesion of the Eastern Poland macroregion, impact on the competitiveness and innovativeness of economy, and impact on the labour market.

Aside from the key issue of the demand for modelling results for cognitive and practical purposes (e.g. the implementation of a formative, conclusive or socio-political function of evaluation in the process of programme or policy management), an essential prerequisite for using macromodelling tools when evaluating the impact of a programme should be access to wide public statistics (cross-sectional, long time series) and, above all, a substantial budget for the development activities under analysis. The amount of planned public resources must be large enough to have a real chance to impact (shock) the recorded macroeconomic values (total Polish GDP or its y/y dynamics, employment/unemployment rate, etc.).

Various modelling tools are used in evaluation. These include both relatively simple single equation econometric models, and advanced multi-equation and general equilibrium models, which assume the reflection of market processes occurring in the whole economy and their interaction, taking into account the dynamics and uncertainty of market economy phenomena. The most advanced tools of this type are DSGE (dynamic stochastic general equilibrium) models (cf. Mogiła et al., 2019; EduMod…, 2015)30. From the evaluation perspective, the key function of a model is to assess the impact that exogenous economic changes and public interventions, as set out by the model user, will have on all modelled variables [...] [a purely evaluative function – author’s note]. An additional purpose of a model is to make conditional forecasts of the future social and economic situation in Poland and to identify factors behind the economic phenomena observed in the past (foresight function) – cf. EduMod..., 2015, p. 11. Referring to the EC’s guidelines concerning the 2000–2006 and 2007–2013 perspectives, we can expect robust macroeconomic models to:

1. reflect the demand and supply structure of the economy, and in the case of the latter to include