Development of a recommender system for Dutch case law, with the use of a topic model.

(1)

Development of a recommender

system for Dutch case law, with the

use of a topic model.

Erwin van den Berg

6353622

Bachelor thesis Credits: 18 EC

Bachelor Opleiding Kunstmatige Intelligentie University of Amsterdam Faculty of Science Science Park 904 1098 XH Amsterdam Supervisors dhr. dr. R.G.F. Winkels dr. dr. A.W.F. Boer Leibniz Center for Law

Faculty of Law University of Amsterdam

Vendelstraat 8 1012 XX Amsterdam

(2)

Summary

Dutch case law is publicly available on rechtspraak.nl, there is however no publicly available recommender system for Dutch case law. The purpose of this project is to give useful suggestions within Dutch case law. These suggestions are generated by comparing the (pre-processed) texts of the case laws using a topic model. The topic model represents the case laws as mixtures of topics, after which the most similar case laws are found by calculating the similarity between the topic mixtures.

The topic similarity based suggestions that this project generated, were evalu-ated by legal experts and novices, who ranked three suggestions from best to worst, and stated whether the suggestion is good enough for a recommender system. One of the three suggestions was based on topic similarity, another was based on refer-ences to legislation and the other was based on a combination of these two methods. Both legal experts and novices showed significant preference for the suggestions based on topic similarity. The legal experts wanted to see 85% of the suggestions based on topic similarity in a recommender system, for the novices this was 86.7%. The legal experts ranked the suggestions based on topic similarity as best sugges-tion 80% of the time, while the novices always ranked the suggessugges-tion based on topic similarity as best suggestion. This suggests that suggestions based on topic similarity can give useful suggestions within Dutch case law.

(3)

Acknowledgements

I would like to thank my supervisors Radboud Winkels and Alexander Boer for assisting me in this project, and for finding legal experts who where willing to evaluate the suggestions.

I would also like to thank all the evaluators who took the time and effort to read many case laws and fill in the survey.

(5)

1 Introduction

Legal information from official sources is easily available nowadays through the Internet. In the Netherlands these official sources are rechtspraak.nl for case laws and wetten.nl for legislation. However, the legal information in these sources is not linked, and there is no suggestion system, which limits the usefulness to users.

This project is part of ongoing research with the goal of making these sources more useful to users by creating a recommender system. In previous research a recommender system was built by linking Dutch legislation based on references within legislation. In later research, Dutch case laws were also taken into account by looking at the references to legislation within the texts. But case laws have never been linked by text comparison, which is what this project focuses on. This leads to the research question: “Is it possible to give useful suggestions, by comparing the content of the text between Dutch case laws (using a topic model)?".

Suggestions within case law based on text rather than on the references to leg-islation can potentially be very useful, because in this way it might be possible to suggest cases that are similar but where different legislation is used.

Before the case laws are compared, they are first pre-processed. The primary purpose of this is to filter out the case laws that lack meaningful information (e.g. case laws that just state that the suspect is found guilty of a certain crime, without context) and to filter out parts of text within case laws that lack meaningful infor-mation (e.g. text about formalities of the process with no relation to the content of the case).

After the pre-processing, the case laws are compared using a topic model, which is an algorithm that discovers the primary themes (topics) within an un-structured collection of texts, and represents the texts as mixtures of these topics. For each case law, the best suggestion is then given by selecting the case law with the most similar topic mixture.

Because it seems probable that legal experts look at case laws differently than novices, it is imaginably that they might evaluate the suggestions within case law differently. To evaluate whether both legal experts and novices will appreciate the given suggestions, four legal experts and three novices are given five randomly selected case laws for which they rank three suggestions from best to worst sug-gestion, and for which they state whether the suggestion is good enough for an recommender system. One of the three suggestions is the most similar according to the used method using a topic model, another is the most relevant based on the references to legislation according to last year’s research (more detail on this in the next chapter, Theoretical foundation), and one of the suggestions is based on a combination of the two methods.

(6)

2 Theoretical foundation

There has not been much research in the linking of Dutch case laws. In previous research, Dutch case laws have been linked by looking at the references (to legis-lation) within the texts. But case laws have never been linked by text comparison. Literature about linking Dutch legislation that does not address Dutch case laws, falls outside the scope of this project because the construction of Dutch case laws is fundamentally different from legislation.

Winkels et al. (2014) [1] describe a way to make references in case law to legislation explicit and machine readable. This is done by extracting the references using regular expressions. They show that by creating a network of references, suggestions to relevant sources of law can be generated. While they had the same intention, namely the creation of a Dutch legal recommender system, the approach is very different from the approach I use. Winkels et al. used references to legisla-tion to link case laws, while I will focus on comparing the text of the case laws.

In order to compare the Dutch case laws, a topic model will be used to find topics in the text of the case laws. Perhaps the most common topic model currently in use is Latent Dirichlet Allocation (LDA) from Blei et al. (2003) [2]. LDA is an unsupervised learning algorithm that initially creates a (by the user specified) num-ber of topics (or clusters) and subsequently represents each text as being a mixture of these topics. The bag of words assumption applies to the LDA algorithm, which means that the algorithm ignores word order and therefor ignores word phrases.

After LDA was published, many extensions of this model were published that do take word order into account (using n-grams). Two examples of this are Wallach (2006) [3] and Wang et al. (2007) [4], they both present extensions of LDA that take word order into account by fitting a more complicated model using unigrams and bigrams. These extensions to LDA could potentially make the topics more meaningful, but they are more compute intensive which is especially undesirable for relatively large corpora such as case laws.

A combination of the efficiency of the LDA model and the usefulness of n-grams is presented by Blei and Lafferty (2009) [5]. In their approach, called Turbo Topics, the topic model is initially fit as usual (in the same way as standard LDA), whereafter significant words that are preceding or succeeding these topic-words are searched in the texts and eventually added to the topic model. With this approach, word phrases are taken into account while maintaining the computational advan-tage and statistical simplicity of unigram topic modeling. This approach seems most suitable for the topic modelling of the Dutch case laws, therefor this approach will be used in this project.

(7)

3 Method

To give suggestions within case law, the case laws were first downloaded from rechtspraak.nl, after which the irrelevant parts were deleted during the pre-processing. Using a topic model, the primary topics of the pre-processed case laws were dis-covered, and the case laws represented as mixtures of these topics. Subsequently, the best suggestion for each case law was found by calculating the similarity be-tween the topic mixtures and selecting the case law with the highest similarity. Eventually, the suggestions from this method were evaluated by legal experts and novices, who compared the suggestions from this method to the suggestions from last year’s method, based on references.

3.1 Domain selection

For the evaluation of this project, the suggestions based on text comparison from this project are compared to the suggestions based on references to legislation of last year’s project by Winkels et al. (2014) [1]. Since last year’s project only used case law from one field of law, namely the foreigners law (‘vreemdelingenrecht’), this project also only focused on that field, so that the results of the two projects could be compared.

Last year’s project has code specific for the foreigners law (to recognize the legislation), but unlike last year’s project, the code for this project is generic and should work on all fields of law.

3.2 Download case laws

The case laws with including metadata are provided by rechtspraak.nl in XML formatted files. The parser form last year’s project1creates XML files that are dif-ferent from the files on rechtspraak.nl. While the XML files on rechtspraak.nl all use the same XML style, last year’s parser creates data with multiple XML nota-tions, which makes it harder to pre-process the data. For this project, a new case law downloader was created (based on last year’s data parser), that can download the unedited case laws from any specified field of law and also from all fields at once. All pre-processing is left out, this makes the data collector easier to use for other purposes, such as future projects.2

The case laws are downloaded to a folder called ‘caselaws’, this folder is cre-ated at the location where the case law downloader is executed. For this project

1

last year’s parser is called ‘parser.py’ and is located at https://github.com/b8vrede/ cite_parser.

2

The case law downloader of this project is called ‘step_1_download_caselaw.py’ and is located at https://github.com/6353622/caselaw_suggestions.

(8)

14443 case laws were downloaded, which were all the case laws within foreigners law on the 26th of May, 2015.

3.3 Pre-process case laws

The pre-processing aims to keep only the relevant parts of the text, therefor it is required to estimate the relevance of the different parts within case laws. Most case laws follow the same format, they are preceded by an indication of what the case is about, the ‘inhoudsindicatie’ followed by the actual case law which con-sists of ‘procesverloop’ (procedure) followed by ‘overwegingen’ (considerations) and ending with ‘beslissing’ (decision). Because the indication part, indicates what the case is about, it is likely that this part is relevant for the recommender system. In many case laws, the procedure part has little relation to the content of the case, instead it is about the formalities of the process, therefore it may not be relevant for a recommendation. The considerations part of the case laws contains the consider-ations with associated references to legislation, and also the decision of the judge, therefor it is likely that this part is relevant for the recommendation. The decision part of the case law does not seem very relevant, because in many case laws it only contains one sentence that was already stated in the considerations part, followed by a few names (of the judge and others).

The first step of the pre-processing checked whether the case laws contain the words ‘inhoudsindicatie’ (contents indication) and ‘overwegingen’ (consider-ations), if a case law did not contain one or both of the words it is moved to a folder called ‘irrelevant’, that folder is ignored in the further processes. The word ‘overwegingen’ was present in 94.3% of the case laws and ‘inhoudsindicatie’ was present in more than 99.9% of the case laws (only 2 of the 14443 case laws had no mention of ‘inhoudsindicatie’). Thus in this first pre-processing step 5.7% of the corpus was transferred to a folder where it will be ignored during the text compar-ison, most of these transferred files are relatively short and do not contain much information.

The next step of the pre-processing filtered out everything except the contents indication part and the considerations part. Selecting the ‘inhoudsindicatie’ (con-tents indication part) was really straightforward thanks to the consistent use of metadata in XML format for this part of the case law. Every ‘inhoudsindicatie’ was preceded by ‘< inhoudsindicatieid =’ and exceeded by ‘< /inhoudsindicatie >’, splitting the file at these strings divided the file in three parts, the first part can be ignored, the second part is the ‘inhoudsindicatie’ and the third part contains the ac-tual case law from where everything except the considerations part will be filtered out. Selecting the considerations part was less straight forward because the meta-data is not always the same for this part. Most of the case laws, 95%, started the considerations part with this string ‘overwegingen <’, these files were split here, the remaining 5% were split at the first occurrence of ‘overwegingen’, this gave

(9)

two parts, whereof the first part can be ignored and the second part is containing the considerations and the decision. To get the decision (‘beslissing’) out of it, the start of this part had to be located. Most of the case laws (over 54%) only had one mention of ‘beslissing’ which is where the decision part starts, the case laws that had more than one mention of ‘beslissing’ almost always start the decision part at the last mention of ‘beslissing’. Therefor, the decision part was filtered out by splitting at the last occurrence of the string ‘beslissing’.

After the parts of the case laws that were considered irrelevant for the recom-mender system are filtered out, there was no use anymore for the XML markup constructs (e.g. </parablock>) and these markup constructs should not be pro-cessed in the text comparison, therefor the XML markup constructs were now re-moved. Since the case laws now longer contain XML code, the extension of the files, ‘.xml’, was changed to ‘.txt’.

Some of the case laws contained very little information, some contained only a few sentences. In order to get rid of these files, the last step of the pre-processing deleted the smallest 1% of the corpus by looking at the file size of all the case laws and then deleting the smallest 1%.3

The evaluation of the pre-processing was done by randomly selecting 25 case laws and checking whether the contents indication and the overwegingen part were selected correctly. The contents indication was correctly selected in all, thanks to the consistent use of XML markup constructs (and the previous deletion of 2 of the 14443 files that had no mention of ‘inhoudsindicatie’). Therefor, the precision and recall was both 100% for this part. The ‘overwegingen’ part was also present in all 25 checked case laws, but because the end of this part is at the last mention of ‘beslissing’, some of the case laws contained a bit of the ‘beslissing’ part in what should be the ‘overwegingen’ part. This happened in 16% of the evaluated case laws.

From the 14443 downloaded case laws in the foreigners law, there were 13481 (93.3%) left for the topic discovery and ultimately for the recommender system.

3.4 Topic discovery in case laws

After the pre-processing, the case laws were compared using a topic model, which is an algorithm that discovers the primary themes (topics) within an unstructured collection of texts, and represents the texts as mixtures of these topics. Perhaps the most common topic model currently in use is Latent Dirichlet Allocation (LDA) from Blei et al. (2003) [2], one downside of this topic model is that it is an bag of words model, which means that it ignores word order and therefor ignores word phrases. A combination of the efficiency of the LDA model and the usefulness

3

The case law pre-processor of this project is called ‘step_2_preprocess_caselaw.py’ and is lo-cated at https://github.com/6353622/caselaw_suggestions.

(10)

of word phrases is presented by Blei and Lafferty (2009) [5]. In their approach, called Turbo Topics, the topic model is initially fit as usual (in the same way as standard LDA), subsequently, significant words that are preceding or succeed-ing these topic-words are searched in the texts and eventually added to the topic model. With this approach, word phrases are taken into account while maintaining the computational advantage and statistical simplicity of unigram topic modeling. This approach seemed most suitable for the topic modelling of the Dutch case laws, therefor this approach is used in this project.

In order to prevent the creation of topics with insignificant words, a stop words list was used which contained words that are common in the Dutch language (e.g. determiners) and words that are common in case law but not very meaningful (e.g. formalities such as salutations). The stop words list was created by initially cre-ating a list of all words that occure in the corpus and subsequently count in how many of the case laws each word occurs, if a word occurs in more than a preset threshold percentage of all case laws, then that word was added to the stop words list. The words in the created stop words list are ordered from highest to lowest oc-currence, thus by looking at the last words in this list it can be determined whether the threshold should be changed.4For this project the treshold was set at 15%, thus words that occur in more than 15% were ignored in the creation of the topics, the created stop words list can be seen in appendix A.

For this project, an open source implementation of LDA and Turbo Topics in Java was used, which is called MALLET (MAchine Learning for LanguagE Toolkit) and is written by McCallum (2002) [6]. Three input arguments that were used in MALLET for this project are: the folder that contains the pre-processed case laws; the created stop words list; and the number of topics that had to be created. For this project the number of topics is set at 150 (more on this in the chapter 7, Discussion and Future Work). The exact MALLET command line argu-ments used for this project can be seen in appendix B.5 Using these commands, MALLET gave two output files, ‘topics.xml’ and ‘composition.txt’. The first file, topics.xml, shows the words and word phrases that the topics consist of, as can be seen in appendix D. The second file, compositions.txt, represents every case law as mixtures of the 150 topics, with this file it is possible to calculate the similarity between case laws.

3.5 Calculation of the suggestions

After the case laws were represented as mixtures of topics, for each case law the best suggestion was found by calculating the similarity between the topic mixtures

4_{The script that created a stop words list is called ‘step_3_stopwords_creator.py’ and is located}

at https://github.com/6353622/caselaw_suggestions.

5

For completeness, the used MALLET command line arguments can also be seen on https: //github.com/6353622/caselaw_suggestions, in ‘step_4_mallet_use.txt’.

(11)

and selecting the case law with the most similar topic mixture. This was done by initially calculating the sum of squared errors between a specified case law and all other case laws, and subsequently converting this sum of squared errors to a similarity value. When the similarities between a specified case law and all other case laws are calculated, a predefined number of suggestions together with their similarity value is given as output, the most similar case law is given first; then the second; and so forth. The similarity value is given as percentage, where 0 % means two case laws have completely different topic mixtures (no overlap), and 99% means two case laws are highly similar.6

An example of the output can be seen in listing 1, where three suggestions were given for case law: ‘ECLI:NL:RBSGR:2009:BH7787’. The output consists of a a list of pairs, the first pair contains the name of the case law for which the suggestions are found (and the similarity with itself, thus 100), and the other pairs are the names of the best, in this case three, suggestions. The advantage of this list that contains all the information as output over printed text as output, is that this list can easily be used in potential future projects (e.g. a future project that creates an interface for the suggestions).

[ ( 1 0 0 , ’ ECLI : NL : RBSGR : 2 0 0 9 : BH7787 ’ ) ,

( 9 9 . 2 8 3 7 8 7 6 3 6 7 4 8 2 7 2 , ’ ECLI : NL : RBSGR : 2 0 0 4 : AQ5970 ’ ) , ( 9 9 . 0 8 8 6 5 8 6 4 1 9 5 9 6 6 8 , ’ ECLI : NL :RBDHA: 2 0 1 5 : 4 9 1 5 ’ ) , ( 9 8 . 6 8 6 0 2 3 0 0 9 2 7 8 2 3 , ’ ECLI : NL : RVS : 2 0 0 4 : AQ5615 ’ ) ]

Listing 1: Output example: three suggestions for

‘ECLI:NL:RBSGR:2009:BH7787’

3.6 Evaluation of the suggestions

Because it seems probable that legal experts look at case laws differently than novices, it is imaginably that they might evaluate the suggestions within case law differently (perhaps the legal experts are more critical than the novices when it comes to the suggestions). The same difference is also imaginable between legal experts that have experience in the used domain, foreigners law, and the legal ex-perts that have no experience in this field of law. To evaluate whether novices, legal experts with experience in foreigners law, and legal experts without experience in the foreigners law would appreciate the given suggestions, people from all three groups were addressed to evaluate the suggestions.

For this project, the evaluators consisted of three novices, two legal experts with experience in the foreigners law, and two legal experts without experience in

6

The script for the calculation of the similarity and the giving of suggestions is called ‘step_5_calculate_similarity_and_give_suggestions.py’ and is located at https://github. com/6353622/caselaw_suggestions.

(12)

this field of law.

The evaluators were given five randomly selected case laws for which they ranked three suggestions from best suggestion to worst, and for which they stated whether any of the three suggestions is good enough for an recommender system. One of the three suggestions was the most similar according to the used method using a topic model, another was the most relevant based on the references to leg-islation according to last year’s research as shown by Winkels et al. (2014) [1] and one of the suggestions was based on a combination of the two methods. For the combination of the two methods, a list of the top 200 suggestions according to the topic model was obtained (for each randomly selected case law), and also a list of the top ten suggestions according to the references to legislation. The first sugges-tion from the top ten list that appeared in the other list of the top 200 suggessugges-tions, was chosen as the suggestion based on both methods.

The interface of last year’s project (that gave suggestions based on legislation) does not give suggestions from case law to case law. However, it was possible to get the most relevant suggestions according to this method out of the network by editing the python script.7

To ensure that the evaluators were unbiased, no information was given to them about the used suggestion methods. They were not aware that three different sug-gestion methods were used. The used evaluation form can be seen in appendix C, which contains an evaluation form that is completed by one of the evaluators.

7

The edited python script to get suggestions within case law based on references is called ‘cre-ate_network_edited.py’ and is located at https://github.com/6353622/references_ network_caselaw_edited.

(13)

4 Results and evaluation

The results from the two legal experts with experience in the foreigners law are merged with the results from the two legal experts without experience in this field of law, in both figure 1 and 2, because the results were almost the same. For exam-ple, all four legal experts, ranked the suggestion based on topic mixtures as best in four of the five case laws (80%).

Figure 1 shows, for each suggestion method, the percentage of suggestions that the evaluators would wanted to see in a recommender system. This figure shows that the legal experts wanted to see 85% of the suggestions based on topic similar-ity in a recommender system. It also shows that novices wanted to see 86.7% of these suggestions in a recommender system.

85% 6,7% 25% 86,7% 0% 6,7% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Topic similarity References Combination

Methods where the suggestions are based on

Legal experts Novices

Figure 1: Percentage of the suggestions from each method, that the evaluators want to see in a recommender system.

Figure 2 shows how often the evaluators ranked each suggestion method as first, second and third best suggestion. This figure shows that the suggestions based on topic simularity are ranked as best suggestion 80% of the time by the legal ex-perts, and 100% of the time by novices.

(14)

Legal experts Novices Legal experts Novices Legal experts Novices Ranked 1st Ranked 2nd Ranked 3rd

Combination 15% 0% 50% 60% 35% 40% References 5% 0% 40% 40% 55% 60% Topic similarity 80% 100% 10% 0% 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Figure 2: Ranking of each suggestion method, presented in percentage of each rank.

Figure 1 and 2 show that the suggestions based on topic similarity are signif-icantly more appreciated than the suggestions based on references and than the suggestions based on a combination of the two, by both legal experts and novices. Figure 1 shows that the legal experts wanted to see 85% of the suggestions based on topic similarity in a recommender system, compared to 6,7% of the suggestions based on references and 25% of the suggestions based on a combination of these two methods. For the novices this difference is even greater, they wanted to see 86.7% of the suggestions based on topic similarity in a recommender system, com-pared to 0% of the suggestions based on references and 6.7% of the suggestions based on a combination of these methods. Figure 2 shows that the legal experts ranked the suggestions based on topic similarity as best suggestion 80% of the time, while they ranked the suggestions based on references only 5% of the time as best, and the suggestions based on the combination of the methods in the remaining 15% of the time. While the novices always ranked the suggestion based on topic similarity as best suggestion.

The suggestions based on the combination of topic similarity and references appear to be more appreciated by legal experts than by novices. This can be seen in figure 1 wich shows that legal experts wanted to see 25% from the suggestions based on both methods in a recommender system, for novices this is no more than 6.7%. That this combination seems to be more appreciated by legal experts can

(15)

also be seen in figure 2, which shows that the legal experts rank the suggestions based on the combination as best suggestion in 15% of the time, while novices never ranked this as best suggestion.

The suggestions based on references alone were in general not appreciated by both the legal experts and the novices, they both ranked more than 50% of these suggestions as worst suggestion (as seen in figure 2) and the novices wanted to see none of these suggestions in a legal recommender system, while the legal experts only wanted to see 6.7% of these suggestions in a recommender system.

(16)

5 Conclusion

Both legal experts and novices showed significant preference for the suggestions based on topic similarity. The legal experts wanted to see 85% of the suggestions based on topic similarity in a recommender system, for the novices this was 86.7%. The legal experts ranked the suggestions based on topic similarity as best sugges-tion 80% of the time, while the novices always ranked the suggessugges-tion based on topic similarity as best suggestion. The results from the evaluation by the legal experts and novices suggest that it is indeed possible to give useful suggestions, by comparing the content of the text between Dutch case laws using a topic model.

The resulting recommender system is only tested on the foreigners law (‘vreemdelin-genrecht’), however, since nothing in the code is written specific for the foreigners law, the recommender system is expected to give similar results when applied on other fields of law within Dutch case law.

6 Discussion and Future Work

The purpose of this project was to give useful suggestions within Dutch case law, and while the results are promising, the suggestions can not yet be obtained in a portal. A new interface could be build for the suggestions of this project, but it might be even more interesting to add these suggestions in the interface that is al-ready build and adjusted in previous project’s, as shown by Winkels et al. (2014) [1].8This interface now only gives suggestions from legislation to legislation, from legislation to case law, and from case law to legislation. This interface does not yet give suggestions from case law to case law, thus implementing the suggestions from this project might be a valuable addition.

Currently the recommendations generated in this project (and most likely the recommendations from all recommender systems within Dutch legal information) are only content-based. Which means that the recommendations are only based on the similarity between the content (namely, the case laws). It would also be inter-esting to base the recommendations on past user behaviour (collaborative filtering) or on the combination of user behaviour and content similarity (Hybrid Recom-mender Systems).

For the topic discovery in this project, the number of topics is set at 150. This number is chosen because less than 100 seemed to low since that is less than one hundredth of the number of case laws (13481). The words that formed one topic when the number of topics was set at 150, were split up in multiple topics when the

8

Last year’s interface is located at https://github.com/b8vrede/new_interface_ wetten.nl.

(17)

number of topics was set at more than 200, and finding the best suggestions take longer when more topics are used. Therefor, for this project 150 is set as the num-ber of topics (primarily to limit the calculation time), but more thoroughly research and evaluation on the number of topics is required. It might even be possible to select the best number of topics automatically, this is however not tested in this project.

Other interesting things to take into account in future research are the date of the case laws, and for which party the decision is (more) positive, in this project this information is not used.

(18)

References

[1] Radboud Winkels, Alexander Boer, Bart Vredebregt, and Alexander van Someren. Towards a legal recommender system. In R. Hoekstra (ed), pages 169–178. JURIX, IOS Press, 2014.

[2] David Blei, Andrew Ng, and Michael Jordan. Latent dirichlet allocation. Jour-nal of Machine Learning Research, 3:993–1022, 2003.

[3] Hanna Wallach. Topic modeling: Beyond bag of words. In Proceedings of the 23rd International Conference on Machine Learning, pages 977–984. ICML, ACM, 2006.

[4] Xuerui Wang, Andrew McCallum, and Xing Wei. Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In ICDM ’07 Proceedings of the 2007 Seventh IEEE International Conference on Data Mining, pages 697–702. ICDM, IEEE Computer Society Washington, 2007. [5] David Blei and John D. Lafferty. Visualizing topics with multi-word

expres-sions. arXiv, 0907, 2009.

[6] Andrew Kachites McCallum. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu, 2002.

(19)

Appendices

A

Stop words list

de van het in dat is een en op heeft niet te voor zijn artikel aan verweerder met eiser worden rechtbank die of bij tot door geen als om vreemdeling lid kan deze naar besluit onder hij uit dit beroep wordt ten zich eerste dan oordeel grond vw ook er aanvraag eiseres uitspraak zij kunnen ter over haar verzoeker hebben verbli-jfsvergunning gesteld indien gelet nederland dient ingevolge nu belang verblijf omstandigheden sprake standpunt zal awb tegen was minister bestreden meer wel voorts toepassing aanhef eisers bedoeld land aanleiding na hem afdeling zou geval staatssecretaris moet tweede bepaalde zoals overwogen volgt zitting a omdat be-handeling staat bewaring verzoek andere hetgeen voorzieningenrechter volgens derhalve beoordeling hierna verklaard zaak verband zover stelt gemaakt nog evrm feiten vreemdelingen tijd situatie maatregel had blijkt gebleken bestaat aangevo-erd uitzetting vraag beleid c moeten aanzien reeds aannemelijk onrechte wet au-toriteiten bezwaar thans april maar brief december hun januari aanmerking b volgende juli punt strijd derde nemen maart onderzoek overwegingen overweegt gegrond betrokken nr komt herkomst juni termijn oktober onvoldoende betrekking november februari waarin recht maken gegeven bescherming daarbij mei gestelde september gronden stellen komen ambtsbericht zonder kosten afgewezen binnen slechts terugkeer kader zin tussen beslissing daartoe aldus algemene hier the vb nationaliteit verlenen orde richtlijn ongegrond informatie genomen vast verleend wijze onderhavige procedure asiel ziet welke voldoende jaar af al terecht augustus ingediend n risico toe feit omstandigheid vc eerder evenmin redelijkheid grondslag zaken hoger verklaringen documenten voorziening doen mogelijk nader gedaan s hiervoor zodat acht wegens beschikking waarvan vergunning proceskosten voor-gaande stukken leiden anders voorlopige f geweest immers verder verdrag d asielaanvraag redenen overgelegd verklaring daarvan waarop gemachtigde arrest alle belangen tijde aangemerkt rechtmatig stelling bestuursrecht tevens genoemde rechten raad periode vastgesteld verlening echter grief bestaan daarom geschil gehoor laten vereiste gezien bezit gemotiveerd rapport openbare persoon per tij-dens voldaan twee geacht conclusie plaats dienen vermeld bijzondere gehouden aard nederlandse buiten zo moment reden vanwege enkele ander hof waarbij maan-den waar nieuwe asielrelaas nieuw vreemdelingenwet stand toelating geven para-graaf overgelegde werd wie wat inzake v eerdere besluiten vragen betreft gegevens vervolgens voren betreffende vertrek mede verwijzing weergegeven beslist drie bevoegdheid kon bepaald verrichten aangegeven kennelijk valt eerst e gaat daar naam algemeen aangevallen motivering re aangezien unie blijkens bepaling mag gaan le leidt jurisprudentie afwijzing datum eg verlaten schadevergoeding eu-ropese neergelegd partijen houden geoordeeld voorwaarden hierop redelijkerwijs beroepschrift vierde rechter noch waarom el personen gevallen omtrent fami-lie gedurende vorenstaande individuele opgenomen gebracht identiteit regufami-lier

(20)

behoren verricht problemen bekend vernietigd aanvragen gevolg beperking ont-breken betoog krachtens mogen beoordeeld zodanig maakt zien proces doel ba-sis mens nationale ljn buitenlandse zelf aanwezig alleen alsnog gekomen ernstige waren state geldt gebruik langer staan dag beginsel daarmee afkomstig justitie ver-leende weg kind faalt toetsing onrechtmatig partij r voormelde regeling dergeli-jke eigen schending and terwijl be asielzoekers beschikt lidstaten vluchteling be-trokkene redelijke gelegd voert wijziging licht gericht bovendien loopt relaas duur tegengeworpen gelegenheid vreemdelingencirculaire sub samenhang vaste docu-ment zicht vanaf veroordeeld doet nadere mogelijkheid to opgelegd genoemd ervan gemaakte betoogt jv veroordelen sinds hierbij ingesteld blijven verwezen verklaren aangevoerde overigens artikelen weken twijfel vormt lande achten plaatsgevonden uitspraken humanitaire betekent weliswaar daarnaast wijst verweerders alsmede enige www mate verschijnen nl bedoelde wanneer zes afgeleid daarop asielzoeker daarin verzocht dagen relevante beoordelen internationale desbetreffende waaruit bepalingen bestuursrechtspraak mr voornemen zouden proceskostenveroordeling slaagt indienen ge gezin voldoet overige rechte concrete treffen aantal gewezen rekening hoofdstuk nadat verblijven aangenomen vergoeding werking feitelijke dus hen bevoegd uur toelichting onderbouwing geweigerd inhoud verstrekt gewor-den onderdeel aanwezigheid gebaseerd verwijst ingang weten europees weiger-ing wegweiger-ingsfactor waaronder vrijheden leven griffier aldaar veiligheid verbleven gehad hoogte rechtsbijstand eveneens der zeer beroepsgronden leggen vereist di-ens deel gerechtvaardigd samengevat klaagt namdi-ens daaraan ambtshalve geldig fundamentele bezien goede verplichting behoort wil voornoemde afweging waarde bedrag beroepsgrond bevestigd name enkel getoetst vreemdelingenbesluit even-wel beroepen jaren middelen tijdig feitelijk zoverre behoeve verklaart ieder hand verstaan zullen terug betekenis geeft hiertoe vier ligt inachtneming gelegen vrij betoogd steeds dezelfde kamer eu opnieuw landen gevaar noodzakelijk laat bij-zonder voormeld voorkomen voorzien vaststelling neemt horen vormen hiervan kennen onderbouwd voet duidelijk geconcludeerd ontvangen ik verantwoordelijk zichzelf kort m hoewel ind gevolgd tenzij uitgangspunt biedt bepalen voldoen groep beantwoording voeren voorhand daarvoor geloofwaardig mee uitgegaan reis zake voorafgaand geheel toen burger overweging positie relevant zowel neder-lands achterwege toegekend l brengt verblijft daartegen volstaan houdt beschouwd ac dossier i uitsluitend griffierecht middel iii kennis zijnde brengen onmiddellijk bieden gebied behoeft bespreking vinden daadwerkelijk direct vijf gevraagd laat-ste schade slagen allereerst vermelde geding voorlopig politie juist enig verwacht korte pas gezegd gevolgen ii hieruit dienst voorwaarde geleid doende overeenkom-stig huis nodig oog afwijking merken doch zie beperkt ze blijft betwist persoonlijk leden taal gegaan lp lopen veroordeling volledig behoeven bevat zorg rol afdoen hoe strekt international welk for krijgen zittingsplaats eeg melden redelijk wijzen p rva eisen afkomst j voort h oud bewijs handelingen minder vallen weer inhoudelijk vrijheid hierom regels dergelijk juiste bestuur gelden overeenkomst voornoemd gebrek merkt g lt gt on

(21)

B

MALLET use

For this project, mallet 2.0.7 was run on a Microsoft Windows 7 machine. Mallet was located at:

C:\mallet\

The downloaded and pre-processed case laws are located at: C:\mallet\step4\caselaws\

The created stop words file was located at: C:\mallet\step4\stopwords.txt

The three used commend line rguments (order must be the same): cd C:\mallet

bin\mallet import-dir --input step4\caselaws --output step4\caselaw.mallet --keep-sequence

--stoplist-file step4\stopwords.txt --token-regex "[\p{L}\p{M}]+" bin\mallet train-topics --input step4\caselaw.mallet

--num-topics 150 --xml-topic-phrase-report step4\topics.xml --output-doc-topics step4\composition.txt

Explanation: --token-regex "[\p{L}\p{M}]+" provides support for unicode letters and marks, without it, only normal letters (a to z) can be used. On a linux machine ’[\p{L}\p{M}]+’should be used.

--xml-phrase-reportprovides support for word phrases, without it, word order is ignored.

(22)

C

Evaluation form

Evaluation form, completed by one of the evaluators: Geachte respondent,

Voor mijn afstudeerproject heb ik, in samenwerking met mijn supervisors Radboud Winkels en Alexander boer, een suggestiesysteem gemaakt voor uitspraken op recht-spraak.nl.

Dit suggestiesysteem geeft bij het lezen van een bepaalde uitspraak, suggesties voor andere mogelijk relevante uitspraken.

Voor de evaluatie van dit suggestiesysteem, zoek ik juristen die voor 5 uitspraken, 3 suggesties willen rangschikken (van goede naar slechte suggestie).

Ik hoop dat u me hierbij wilt helpen.

Hieronder volgt eerst een antwoord voorbeeld, gevolgd door de uitspraken met de te rangschikken suggesties.

Voorbeeld antwoorden:

De uitspraak, waarvoor suggesties gegeven worden: www.link-voorbeeld.nl/link-voorbeeld

Rangschik onderstaande 3 uitspraken door een cijfer van 1 t/m 3 erachter te zetten. (Hierbij staat 1 voor de beste suggestie en 3 voor de slechtste suggestie.)

www.link-voorbeeld.nl/link-voorbeeldA 2 www.link-voorbeeld.nl/link-voorbeeldB 3 www.link-voorbeeld.nl/link-voorbeeldC 1

Welke van de 3 uitspraken zou u terug willen zien in een suggestiesysteem? Waarom? (U kunt verwijzen naar de uitspraken a.d.h.v. uw rangschikking.)

Voorbeeld antwoord: Geen één/alleen 1/ 1 & 2/ alle 3, omdat.. Eventuele overige opmerkingen:

...

Hieronder volgen de te evalueren uitspraken met suggesties. De uitspraken zijn allemaal binnen het vreemdelingenrecht. Heeft u ervaring met vreemdelingenrecht?

De 1e uitspraak, waarvoor suggesties gegeven worden:

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2009:BH7787 1F/personal knowing and participation/Turkije PKK/3 EVRM

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2012:BV1971 1 1F/per-sonal knowing participation/Armenië/3EVRM

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2004:AQ5970 2 1F/Turkije PKK/kindsoldaten

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2007:BA6595 3 3/EVRM alleenstaande vrouwen Irak

(23)

Eventuele overige opmerkingen:

rangschikken wordt gecompliceerd door feit dat 1e uitspraak twee rechtsvragen bevat (uit-sluiting 1F VV en uitzetting ism 3 EVRM). Uspr vooral interessant vanwege 1F kwestie en personal knowing and participation toets. Daar gaat uspr 1 ook ihbz over. Uitspraak 2 gaat weliswaar ook over 1F en PKK, maar betreft kindsoldatenkwestie-andere rechtsvraag. De 2e uitspraak, waarvoor suggesties gegeven worden:

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2012:BX3443

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2002:AE5237 3 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RVS:2009:BI4791 2 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBDHA:2013:14533 1

Alleen 1. Gaat ook over tamils en gewijzigd recht. 3 gaat over totaal andere kwestie (medische behandeling). 2 is bodemuspr ELgafaji arrest HvJEU, slechts zijdelings relevant.

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBZWB:2013:3140

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2006:AW6891 1 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2005:AU1829 2 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2002:AF5172 3

Alleen 1. Gaat ook over vervolging op grond van religie Pakistan. Andere twee uspr betreft andere kwesties – afschaffen categoriaal beleid en een kunstenaar uit Pakistan. Eventuele overige opmerkingen:

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RVS:2014:1575

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2008:BF3742 2 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RVS:2015:1035 1 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2010:BO4782 3

alleen 1. Gaat ook over vrouwenbesnijdenis Nigeria. Twee andere over medisch uitzetbeletsel/beschikbaarheid behandeling in land van herkomst

(24)

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RVS:2011:BQ8488

http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2002:AE7009 1 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2004:AO9554 3 http://deeplink.rechtspraak.nl/uitspraak?id=ECLI:NL:RBSGR:2010:BN8682 2

alleen 1. Gaat ook over 3 evrm/minderheidsclan Somalie. Andere gaan over algemeen risico Irak en beschikbaarheid vluchtalternatief Pakistan

Eventuele overige opmerkingen: Bedankt voor uw hulp!

Met vriendelijke groet, Erwin van den Berg

(25)

D

Topics

The topics for the case laws (deleted some XML markup constructs for readability):

<?xml version="1.0"?>

<topic id="0">mobiele telefoon, onverwijld gemeld, uitlezen mobiele telefoon, gehoord, advocaat, raadsman, geschaad, bijstand, wenst, spoedig</topic>

<topic id="1">gelijke strekking, bestuurlijke fase, ne bis idem, herhaalde nova, ve-randerde, herhaalde veve-randerde, vloeit afwijzend gelijke strekking instellen bereikt bestuursrechter toetst ware, herhaalde herhaalde, bestuurlijke fase veranderde voorgedaan bestuursrechter, gelijke strekking ingestelde beoordelingskader</topic>

<topic id="2">manifest bedrog, verlengde asielprocedure, voorzitter staten generaal, rust voorbereidingstermijn, bondsrepubliek duitsland, duitse, overleg, misbruik, vanuit, bijvoorbeeld</topic>

<topic id="3">hongarije, hongaarse, und, unhcr, von, des, werden, gedetineerd, pro, zu</topic>

<topic id="4">coa opvang, opvang coa, verstrekkingen categorieën, opvang geboden, coa coa, verstrekkingen coa, orgaan opvang, opvang opvang, centraal orgaan opvang coa, continuering opvang</topic>

<topic id="5">aanmeldcentrum schiphol, uren, uitgereikt, aanvang, beschikbaar, pro-cesuren, uurstermijn, aangevangen, rechtshulpverlener, uitreiking</topic>

<topic id="6">rechtsmiddel aangewend, rechtsmiddelen aangewend, ingestelde ont-vankelijk, procesbelang doorprocederen, ontvankelijk procesbelang, procesbelang ontvankelijk, ontvankelijk ontvankelijk, rechtsmiddel open, bepaaldelijk gevolmachtigd, publiekrechtelijke rechtshandeling</topic>

<topic id="7">hezb wahdat, functie, werkzaamheden, militaire, regime, mensenrechten-schendingen, politieke, dvpa, comité, commandant</topic>

<topic id="8">mening, meent, namelijk, niets, degelijk, altijd, erop, desgevraagd, nooit, eens</topic>

<topic id="9">gewapend conflict, binnenlands gewapend conflict, willekeurig geweld, internationaal binnenlands gewapend conflict, definitierichtlijn definitierichtlijn, uitzon-derlijke definitierichtlijn, minimumnormen erkenning, willekeurig geweld internationaal binnenlands gewapend conflict, hoog zwaarwegende, subsidiaire beschermingssta-tus</topic>

<topic id="10">gedingstukken verhandelde, toevoeging betaling, toevoeging betaling geschieden, termen behorende bijlage begroot totaal beroepsmatig, gestorte bedrage, mitsdien, x, bpb, één, doorgebracht</topic>

<topic id="11">verschuldigde leges, langdurig ingezetene, betalen leges, leges ver-schuldigd, status langdurig ingezetene, restitutie leges, leges betaald, leges geheven, leges betalen, betaling leges</topic>

<topic id="12">positieve overtuigingskracht, staving identiteitspapieren bescheiden overleggen bescheiden rekenen, niveau bijzonderheden, hetzij hetzij, positieve overtuig-ingskracht uitgaat, positieve overtuigovertuig-ingskracht ontbeert, toerekenbaar ongedocumenteerd, toerekenbaar reisroute, rechtsgrond staving identiteitspapieren bescheiden overleggen bescheiden rekenen, ongerijmde wendingen</topic>

<topic id="13">personalia opgegeven, opgegeven leeftijd, leeftijdsonderzoek, geboorte-datum, verschillende, geboren, resultaten, verslag, meerderjarig, ouder</topic>

<topic id="14">détournement pouvoir, uitstel uitstel, opleggen lichter, uitstel vertrek-beleid, maatregelen, toegepast, vrijheidsbeperkende, meldplicht, opgelegde,

(26)

opleg-ging</topic>

<topic id="15">parlementaire geschiedenis, bedoeling wetgever, uitleg begrip, geschiede-nis totstandkoming, invulling begrip, wetgever beoogd, kamerstukken blz, definitie begrip, memorie tk, voorgestane uitleg</topic>

<topic id="16">vernietiging opwerpt rechtseenheid rechtsontwikkeling rechts-bescherming, aanleg voorgedragen, aangehecht rechtsvoorganger, immigratie integratie, opgeworpen rechtsvraag, uitdrukkelijk voorbehoud, aangehecht rechtsvoorgangers, hoger-beroepschrift vernietiging opwerpt rechtseenheid rechtsontwikkeling rechtsbescherming, immigratie aangehecht rechtsvoorgangers, opgeworpen rechtsvraag beantwoord</topic> <topic id="17">vreemdelingenzaken integratie, kennisgeving, uiterlijk, ontvangst, hoofd, behandeld, mandaat, immigratie, gesloten, stcrt</topic>

<topic id="18">vermoeden illegaal, verbaal staandehouding, verbaal bevindingen, wij verbalisanten, objectieve maatstaven gemeten vermoeden illegaal, processen verbaal, woning, binnentreden, controle, toestemming</topic>

<topic id="19">bekering christendom, bekeerd christendom, geloof, geloofsovertuiging, kerk, gedragslijn, motieven, christelijke, ongeloofwaardig, doop</topic>

<topic id="20">dochter dochter, hoor wederhoor, jongste dochter, contact, stichting, dochters, omgangsregeling, moeder, gemeente, vriendin</topic>

<topic id="21">contra expertise, contra expertise taalanalyse, opsteller contra expertise, uitkomst taalanalyse, eenduidig herleiden spraak cultuurgemeenschap, contra expertise uitvoeren, taalanalyse contra expertise, contra expertise besluitvorming, contra expertise taalstudio, contra expertises</topic>

<topic id="22">appellant appellant, rijkswet nederlanderschap, rwn, handleiding, natural-isatie, voorafgaande, verblijfsgat, indiening, aruba, vermoedens</topic>

<topic id="23">ex tunc, geschreven ongeschreven rechtsregels doorstaan, ex nunc, zulks bovengenoemde, toevoeging betaling, ongegrondverklaring geschreven ongeschreven rechtsregels doorstaan, toevoeging betaling geschieden, casu, mitsdien, terzake</topic> <topic id="24">art art, adversarial proceedings, art rb, aivd, onderliggende, inzage, inlichtingen, concreet, kennisneming, besluitvorming</topic>

<topic id="25">schuld vertrekken, diplomatieke vertegenwoordiging, consulaire verte-genwoordiging, diplomatieke consulaire verteverte-genwoordiging, pogingen ondernomen, vervangend reisdocument, consulaat generaal, ambassade brussel, verkrijgen, er-itrea</topic>

<topic id="26">aandacht besteed, meest recente, recente ontwikkelingen, aandacht besteden, huidige, abrvs, mogelijke, specifieke, verleden, zijde</topic>

<topic id="27">aanvullend protocol, arbeid zelfstandige, wezenlijk gediend, advies economische, criterium, onderneming, ondernemingsplan, turkse, standstill, puntensys-teem</topic>

<topic id="28">echtgenote echtgenote, huwelijk getreden, verbreking huwelijk, huwelijk echtgenote, gesloten huwelijk, huwelijk gesloten, voortgezet, gehuwd, vrouw, re-latie</topic>

<topic id="29">bewijsnood verkeert, gelegaliseerde geboorteakte, reisdocument, buitenlands, overleggen, verkrijgen, nederlanderschap, handleiding, paspoort, aange-toond</topic>

<topic id="30">werkzaamheden zelfstandigen, gehandhaafd kamerstukken, mogendheden verbindend volkenrechtelijke organisatie, bedrijf, bulgarije, werknemers, verkeer, vweu, bulgaarse, bijlage</topic>

<topic id="31">psychische klachten, psychische gesteldheid, traumatische ervaringen, psychische problematiek, psychische toestand, posttraumatische stressstoornis, veilig

(27)

ervaren, posttraumatische stressstoornis ptss, medische, arts</topic>

<topic id="32">bekeerd christendom, bekering christendom, iran iran, refugié sur place, koptische christenen, bekeerde christenen, iraanse, geloof, vervolging, moslims</topic> <topic id="33">brand gestoken, man, vriend, moest, mannen, mensen, kwam, wilde, woning, auto</topic>

<topic id="34">beantwoordt bevestigend, beantwoordt ontkennend, ontkennend beantwo-ord, bevestigend beantwobeantwo-ord, vreemdelinge vreemdelinge, criterium, rechtsoverweging, beide, gehanteerde, criteria</topic>

<topic id="35">khad wad, unhcr note, significante uitzondering, afghanistan, bronnen, vluchtelingenverdrag, afghaanse, officieren, ministerie, juistheid</topic>

<topic id="36">herhaald ingelast, zienswijze, ingegaan zienswijze, zienswijze zienswijze, zienswijze schriftelijk, onzorgvuldig gehandeld, libië, aanvullende, voornemens, rea-geren</topic>

<topic id="37">aanleggen handboeien, gerecht miskend, tijdelijk, tijdelijke, gevoerde, india, ingestelde, ltu, antillen, toegelaten</topic>

<topic id="38">mocht vertrouwen, vertrouwen gewekt, vertrouwen ontlenen, gedane toezegging, vertrouwensbeginsel, erop, hieraan, colombia, moest, nadien</topic>

<topic id="39">verantwoordelijke lidstaat, lidstaat asielverzoek, dublin claim, verorden-ing verordenverorden-ing, verordenverorden-ing asielverzoek, bindend volkenrechtelijke, dublin verordenverorden-ing, asielverzoek behandelen, overdracht, overname</topic>

<topic id="40">kunt u, dactyloscopisch signalement, mijn, heb, pagina, vingerafdrukken, uw, ben, goed, afgenomen</topic>

<topic id="41">onderworpen folteringen onmenselijke vernederende behandelingen bestraffingen, hetzij hetzij rechtsgrond, gegronde vrees, verdragsvluchteling gegronde, verlangd terugkeert, reëel onderworpen, reëel onderworpen folteringen onmenselijke vernederende behandelingen bestraffingen, niemand onderworpen folteringen on-menselijke vernederende behandelingen bestraffingen, vrees vervolging, gegronde vrees vervolging</topic>

<topic id="42">human rights watch, verslechtering veiligheidssituatie, verslechterde veiligheidssituatie, united nations, unhcr eligibility guidelines, unhcr guidelines, united nations high commissioner refugees unhcr, general violence, verenigde naties, uitzonder-lijke</topic>

<topic id="43">militaire dienstplicht, oproep militaire, dienstplicht vervullen, bestraffing, oekraïne, dienstweigering, desertie, leger, opgeroepen, ingezet</topic>

<topic id="44">paspoort paspoort, nationaal paspoort, vals paspoort, kopie paspoort, verlopen paspoort, identiteitskaart paspoort, paspoort afgegeven, regering onderdaan grensoverschrijding, paspoort verkrijgen, geldige identiteitskaart paspoort</topic> <topic id="45">inmenging gezinsleven, objectieve belemmering gezinsleven, gezinsleven oefenen, eerbiediging gezinsleven, uitoefenen gezinsleven, uitoefening gezinsleven, objectieve belemmeringen gezinsleven, respect gezinsleven, family life, objectieve belemmering gezinsleven oefenen</topic>

<topic id="46">koninklijke marechaussee, koninklijke marechaussee kmar, vals bevon-den, authenticiteit, kopie, bureau, originele, afgegeven, identiteitskaart, nepal</topic> <topic id="47">voortvarend gehandeld, voortvarend handelen, laissez passer, voort-varendheid betracht, voortvarend handelt, voortvarend gewerkt, vertrekgesprek gevoerd, handelt voortvarend, voortduren vrijheidsontneming, voortvarend werkt</topic>

<topic id="48">beëindiging verstrekkingen, beëindigd, last, vordering, beëindigen, werkwijze, mededeling, stappenplan, voorzieningen, medewerking</topic>

(28)

verzet afdoening, opgekomen procesorde verzet afdoening daardoor ontoelaatbaar vertraagd, appellanten, gewijzigde, herziening, voorzover, vernietiging, waarmee</topic> <topic id="50">onttrokken toezicht, rechtvaardiging onderscheid, kinderen, over-gangsregeling, beeld, langdurig, ivbpr, kinderpardonregeling, ouders, instanties</topic> <topic id="51">bedrijf, appellante, werknemers, onderneming, werkzaamheden, wav, arbeid, gevestigd, vweu, republiek</topic>

<topic id="52">human rights, from, by, have, as, are, were, or, at, been</topic>

<topic id="53">verenigde staten, surinaamse onderdanen, benelux zwitserland, suriname, bijlage, handel, ontlenen, zwitserse, beide, koninkrijk</topic>

<topic id="54">saoedi arabië, objectieve bron, objectief verifieerbare, ononderbroken woonplaats, aangetoond, brieven, heer, getuige, getuigen, tonen</topic>

<topic id="55">bosnië herzegovina, beginselen behoorlijk, enerzijds anderzijds, algerije, algerijnse, verkeerde, mening, veronderstelling, tbs, willen</topic>

<topic id="56">wettelijk voorschrift, wettelijk vertegenwoordiger, voorschrift vv, verzuim herstellen, herstel verzuim, verbindend voorschrift, aanvrager, bestuursorgaan, bescheiden, vullen</topic>

<topic id="57">shabaab, woonplaats, plaatsnaam, handhaven, controle, omgeving, boord, schip, ervaring, westen</topic>

<topic id="58">penitentiaire inrichting, reglement regime grenslogies, detentieboot reno, tenuitvoerlegging, ruimte, schiphol, politiecel, uitzetcentrum, overgeplaatst, rotter-dam</topic>

<topic id="59">arbeid loondienst, sociale bijstand, duurzaam zelfstandig, zelfstandig duurzaam, regelmatige inkomsten, stelsel sociale bijstand, referent, inkomen, middelen-vereiste, uitkering</topic>

<topic id="60">discriminatoire bejegening, syrië, discriminatie, beschikkingen, syrische, ondervonden, school, hadden, onmogelijk, ernstig</topic>

<topic id="61">democratische republiek congo drc, congo brazzaville, rcd goma, bundu dia kongo, democratic republic congo, strafhof, icc, congolese, kinshasa, fni</topic> <topic id="62">nevenzittingsplaats amsterdam, nevenzittingsplaats zwolle, neven-zittingsplaats haarlem, nevenneven-zittingsplaats assen, nevenneven-zittingsplaats hertogenbosch, nevenzittingsplaats almelo, nevenzittingsplaats arnhem, nevenzittingsplaats groningen, nevenzittingsplaats utrecht, gravenhage nevenzittingsplaats</topic>

<topic id="63">vooruitzicht verwijdering, vlucht geboekt, gedwongen uitzettingen, gedwongen verwijdering, removal order, memorandum understanding, memorandum understanding mou, gedwongen uitgezet, via, zetten</topic>

<topic id="64">genitale verminking, slachtoffer mensenhandel, slachtoffers mensen-handel, aangifte mensenmensen-handel, nigeria, liberia, besnijdenis, nigeriaanse, liberiaanse, besneden</topic>

<topic id="65">lijdt ziekte vergevorderd levensbedreigend stadium, st kitts, uitzonderlijke dwingende, vergevorderd levensbedreigend stadium ongeneeslijke ziekte, hiv infectie, uitzonderlijke lijdt ziekte vergevorderd levensbedreigend stadium, medische, ghana, medicatie, beschikbaar</topic>

<topic id="66">dt amp, amp amp, amp amp amp, amp target blank href http raadvanstate zoeken zoekresultaat verdict id, amp small amp, amp dt amp, amp target blank href http raadvanstate zoeken zoekresultaat verdict id cy firkmu y amp amp amp, target blank href http raadvanstate zoeken zoekresultaat verdict id qytp ymoxx amp amp amp, amp target blank href http raadvanstate zoeken zoekresultaat pdf download asp verdict filename pdf amp amp amp, ro minjus resultpage aspx snelzoeken true amp searchtype amp</topic> <topic id="67">rechterlijke toets, toets doorstaan, rechterlijke toets doorstaan, primair,

(29)

subsidiair, toetsen, neergelegde, ingenomen, oordelen, verweerschrift</topic>

<topic id="68">zag, zei, omstreeks, adres, moest, verdachte, mij, wilde, verbalisant, vroeg</topic>

<topic id="69">president president, fungerend president, openbaar uitgesproken, arrondissementsrechtbank gravenhage, openbaar ministerie, uitgesproken openbaar, afschrift, nederlanden, rechtsmiddel, vreemdelingenzaken</topic>

<topic id="70">sri lanka, sri lankaanse, tamils sri lanka, noorden sri lanka, ltte, littekens, colombo, srilankaanse, risicofactoren, torture</topic>

<topic id="71">inreisverbod uitgevaardigd, opleggen inreisverbod, terugkeerbesluit inreisverbod, uitvaardigen inreisverbod, inreisverbod inreisverbod, uitgevaardigde in-reisverbod, opgelegde inin-reisverbod, uitvaardiging inin-reisverbod, inreisverbod vaardigen, oplegging inreisverbod</topic>

<topic id="72">positieve overtuigingskracht, tegenstrijdige afgelegd, correcties aanvullin-gen, afbreuk geloofwaardigheid, positieve overtuigingskracht mist, innerlijk tegenstrijdig, positieve overtuigingskracht uitgaat, positieve overtuigingskracht uitgaan, geconstateerde tegenstrijdigheden, positieve overtuigingskracht ontbeert</topic>

<topic id="73">gezinsband verbroken, verruimde gezinshereniging, eiseressen eiseressen, referent referent, gezinsband referent, machtiging mvv, gezinsband referente, family life, referente referente, gezinshereniging gezinsvorming</topic>

<topic id="74">termijnoverschrijding verschoonbaar, verschoonbare termijnoverschri-jding, aangetekend verzonden, adres, bezwaarschrift, verzending, bekendmaking, ontvankelijk, ontvangst, post</topic>

<topic id="75">opheffing ongewenstverklaring, glijdende schaal, ongewenstverklaring ongewenstverklaring, ongewenst verklaarde, wetboek strafrecht, ongewenst ongewen-stverklaring, ongewenstverklaring opgeheven, onvoorwaardelijke gevangenisstraf, plegen misdrijf, strafbare gepleegd</topic>

<topic id="76">duurzaam verzet, tegenwerping vluchtelingenverdrag, blijvend onthouden disproportioneel, toepasselijkheid vluchtelingenverdrag, uitzonderlijke bevindt, uitgezet, tien, vertrekplicht, verblijfstitel, betrekkingen</topic>

<topic id="77">voorbereiding verwijderingsprocedure ontwijkt belemmert, terug-keerbesluit terugterug-keerbesluit, terugkeerrichtlijn, terugkeerrichtlijn terugkeerrichtlijn, terugkeerbesluit terugkeerrichtlijn, ontwijken belemmeren, gemeenschappelijke normen, vooruitzicht verwijdering, implementatie terugkeerrichtlijn, parlement gemeenschappeli-jke normen</topic>

<topic id="78">laissez passer, afgifte laissez passer, laissez passer afgegeven, laissez passers, medewerking verleent, laissez passer verstrekken, chinese, laissez passeraanvraag, laissez passertraject, laissez passeraanvragen</topic>

<topic id="79">bma advies, advies bma, bma adviezen, bma arts, advisering bma, veilige behandelomgeving, advies bureau, medische noodsituatie, advies advies, adviezen bma</topic>

<topic id="80">woon verblijfplaats, toezicht onttrekken, vermoeden onttrekken, beveelt opheffing, inbewaringstelling inbewaringstelling, toezicht onttrokken, zulks vordert, onttrekking toezicht, onttrekken toezicht, inbewaringstelling opgeheven</topic>

<topic id="81">overdracht italië, interstatelijk vertrouwensbeginsel, italië overgedragen, italiaanse, ehrm, strijdig, verordening, aspecten, asielprocedure, lidstaat</topic>

<topic id="82">actuele bedreiging, actuele werkelijke bedreiging fundamenteel samen-leving, gedrag actuele werkelijke bedreiging fundamenteel samensamen-leving, persoonlijke, maatregelen, strafrechtelijke, veroordelingen, verblijfsbeëindiging, verblijfsrecht, gepleegd</topic>

(30)

<topic id="83">deelgenomen demonstratie, falun gong, china, chinese, buitenland, foto, activiteiten, deelname, oeigoeren, website</topic>

<topic id="84">noord irak, centraal irak, centraal iraakse, irak irak, irak iraakse, saddam hoessein, centraal irakezen, kdp puk, noord irak noord irak, puk kdp</topic>

<topic id="85">verenigd koninkrijk, interim measure, interim measures, echr coe int, getroffen interim measure, arresten ehrm, interim measure getroffen, echr coe int echr, ehrm ehrm, ehrm interim measure</topic>

<topic id="86">afghanistan, afghanistan afghanistan, westerse levensstijl, hazara afghanistan, mazar sharif, vrouwen afghanistan, taliban taliban, afghaanse, pakistan, kabul</topic>

<topic id="87">inherente afwijkingsbevoegdheid, beleidsregel doelen, vreemdelingenza-ken integratie, gelijkheidsbeginsel, discretionaire, reguliere, schrijnende, factoren, motie, asielgerelateerde</topic>

<topic id="88">strafrechtelijke detentie, criminele antecedenten, belangenafweging voordeel, zwaarder wegen, gediende verhouding ernst daardoor geschonden, zwaarder weegt, belangenafweging voordeel uitvallen, strafrechtelijke antecedenten, kenbare belangenafweging, zware criminele antecedenten</topic>

<topic id="89">binnenlands vestigingsalternatief, binnenlands vluchtalternatief, vlucht vestigingsalternatief, onttrekken elders, vestigen, verblijfsalternatief, banden, gebieden, veilig, tegenwerpen</topic>

<topic id="90">appellante appellante, geschiedenis totstandkoming, werkgever, dagblad, maatregelen, illegale, boetes, vennootschappen, doelstellingen, tewerkstelling</topic> <topic id="91">terugwerkende kracht, intrekking onbepaalde, onbepaalde ingetrokken, terugwerkende kracht ingetrokken, intrekking terugwerkende kracht, intrekking intrekking, verlengen geldigheidsduur, intrekking verblijfsvergunningen, omgezet on-bepaalde, verblijfsvergunningen ingetrokken</topic>

<topic id="92">vrezen vervolging, vrees vervolging, negatieve belangstelling, vervolging vrezen, gegronde vrees vervolging, gegronde vrezen vervolging, vooropgesteld politieke mensenrechtensituatie, gegronde vrezen vervolging vluchtelingenrechtelijke, vermoeden wekken, genève status vluchtelingen</topic>

<topic id="93">turkije, turkse, pkk, soedan, acv, soedanese, nuba, sudan, teruggekeerd, arabische</topic>

<topic id="94">zorgvuldig voorbereid, onzorgvuldig voorbereid, klemt temeer, ont-beert deugdelijke, berust deugdelijke, deugdelijke ontont-beert, zorgvuldige voorbereiding, draagkrachtige ontbeert, zorgvuldigheid voorbereid, ontbeert draagkrachtige</topic> <topic id="95">verstrekken onjuiste, onjuiste achtergehouden, onjuiste onvolledige, onjuiste lezing, onjuiste achtergehouden oorspronkelijke, ambtelijke misslag, verstrekken onjuiste achterhouden, verkregen, berust, valse</topic>

<topic id="96">ecli rvs, ecli xx, vernietigt draagt, veroordeelt beroepsmatig, vernietigt bepaalt, geboren geboortedag, veroordeelt betalen, vernietigt draagt verzending, palestij-nen libanon, vernietigt bepaalt veroordeelt</topic>

<topic id="97">werkgever wav, opgelegde boete, overtreding wav, wav overtreden, uwv werkbedrijf, boeterapport gevoegde, boete overtreding, matiging boete, arbeid wav, boete overtreding arbeid wav</topic>

<topic id="98">onbekende bestemming vertrokken, telefonisch contact, telefonische mededeling, schriftelijke reactie, meegedeeld, bericht, medewerker, schrijven, faxbericht, stuk</topic>

<topic id="99">prioriteitgenietend aanbod, arbeidsmarktaantekening arbeid, arbeid wav, arbeid loondienst, tewerkstellingsvergunning twv, uwv werkbedrijf, werkgever,

(31)

beschikbaar, functie, uitvoeringsregels</topic>

<topic id="100">groot brittannië, vrijheidsontnemende ex, voortzetting, voortduring, afdoening, gegrondverklaring, voortgezet, bijdragen, afgedaan, afwijzende</topic> <topic id="101">vrijstelling mvv, geldige mvv, vrijgesteld mvv, onbillijkheid overwe-gende, machtiging mvv, mvv hardheidsclausule, geldige machtiging mvv, mvv vrijgesteld, mvv mvv, mvv onbillijkheid overwegende</topic>

<topic id="102">den haag, geregistreerd nummer, verbieden totdat, den bosch, gereg-istreerd nummers, o t, w, heden, beide, hetzelfde</topic>

<topic id="103">zuid centraal somalië, zuid somalië, centraal zuid somalië, sufi elmi, zuid afrika, zuid afrikaanse, luchthaven mogadishu, shabaab, uitzonderlijke, defini-tierichtlijn</topic>

<topic id="104">mtv controle, toezicht, controles, uitgevoerd, illegaal, effect, staande-houding, hetzelfde, uitoefening, grensoverschrijding</topic>

<topic id="105">overbrenging ophouding, strafrechtelijk voortraject, wetboek strafrecht, strafrechtelijke aanhouding, wetboek strafvordering, overgebracht bestemd verhoor, strafrechtelijk aangehouden, inbewaringstelling, opgehouden, bevel</topic>

<topic id="106">volgen studie, opleiding, overeenstemming, gelaten, wetgeving, tegen-over, toetsen, rechtstreeks, mocht, weigert</topic>

<topic id="107">kwetsbare minderheidsgroep, reëel strijdige, alleenstaande vrouwen, salah sheekh, seksueel geweld, systematisch blootgesteld, vrouwen meisjes, reëel onder-worpen, ehrm, specifieke</topic>

<topic id="108">minderjarige kinderen, more than normal emotional ties, jonge kinderen, kinderen geboren, onevenredige hardheid, geboren kinderen, kinderen kinderen, meerder-jarige kinderen, jonge leeftijd, ouders kinderen</topic>

<topic id="109">vader moeder, vader broer, vader vader, broer zus, broer broer, dood vader, moeder moeder, moeder zus, overlijden vader, moeder broer</topic>

<topic id="110">wijzigingsbesluit wbv, contra indicatie, wbv wbv, generaal pardon, afwikkeling nalatenschap, afwikkeling nalatenschap oude, contra indicaties, aanbod, ononderbroken, pardonregeling</topic>

<topic id="111">echtgenoot echtgenoot, servië montenegro, signalering sis, voormalig echtgenoot, sis signalering, kosovo, servische, joegoslavië, republiek, albanese</topic> <topic id="112">dwangsom beslissen, gelegaliseerde geverifieerde, beslistermijn, legal-isatie, wettelijke, gebreke, ongehuwd, officiële, verstreken, aangetoond</topic>

<topic id="113">juistheid volledigheid, aanknopingspunten juistheid volledigheid, on-partijdige objectieve inzichtelijke, concreet aanknopingspunt, aanknopingspunt juistheid, uitgaan aanknopingspunten juistheid volledigheid, objectieve bronnen, juistheid uitgaan aanknopingspunten juistheid volledigheid, individueel, ambtsberichten</topic>

<topic id="114">afgifte visum, sociale binding, economische binding, referent, gewaar-borgd, voorgenomen, tijdige, toegang, visa, visumcode</topic>

<topic id="115">hoofdzaak onverwijlde spoed, bijdragen hoofdzaak, verzoekers verzoek-ers, hoofdzaak onverwijlde spoed hoofdzaak, hoofdzaak administratief onverwijlde spoed, administratief hoofdzaak onverwijlde spoed, hoofdzaak onverwijlde spoed hoofdzaak bijdragen hoofdzaak, getroffen onverwijlde spoed, opgeschort totdat, hoofdzaak uitnodig-ing</topic>

<topic id="116">verzoekster verzoekster, onverwijlde spoed, hoofdzaak onverwijlde spoed, bijdragen hoofdzaak, verzoeksters, kans, verzoekschrift, toewijzing, administratief, toegewezen</topic>

<topic id="117">reële daadwerkelijke, afgifte gemeenschapsonderdaan, onderdaan lidstaat, oprichting gemeenschap, verblijfsrecht gemeenschapsonderdaan, grondgebied

Development of a recommender system for Dutch case law, with the use of a topic model.