2nd Workshop on Evaluation and Experimental Design in Data Mining and Machine Learning (EDML 2020) @ ECML PKDD 2020
EDML 2020
- URL: https://imada.sdu.dk/Research/EDML/2020/
- Event Date: 2020-09-14 ~ 2020-09-18
- Submission Date: 2020-06-09
- Location: Ghent, Belgium
*Description*
A vital part of proposing new machine learning and data mining approaches is evaluating them empirically to allow an assessment of their capabilities. Numerous choices go into setting up such experiments: how to choose the data, how to preprocess them (or not), potential problems associated with the selection of datasets, what other techniques to compare to (if any), what metrics to evaluate, etc. and last but not least how to present and interpret the results. Learning how to make those choices on-the-job, often by copying the evaluation protocols used in the existing literature, can easily lead to the development of problematic habits. Numerous, albeit scattered, publications have called attention to those questions and have occasionally called into question published results, or the usability of published methods. At a time of intense discussions about a reproducibility crisis in natural, social, and life sciences, and conferences such as SIGMOD, KDD, and ECML/PKDD encouraging researchers to make their work as reproducible as possible, we therefore feel that it is important to bring researchers together, and discuss those issues on a fundamental level.
A related issue is directly related to the first choice mentioned above: even the best-designed experiment carries only limited information if the underlying data are lacking. We therefore also want to discuss questions related to the availability of data, whether they are reliable, diverse, and whether they correspond realistic and/or challenging problem settings.
*Topics*
In this workshop, we mainly solicit contributions that discuss those questions on a fundamental level, take stock of the state-of-the-art, offer theoretical arguments, or take well-argued positions, as well as actual evaluation papers that offer new insights, e.g. question published results, or shine the spotlight on the characteristics of existing benchmark data sets.
As such, topics include, but are not limited to:
- Benchmark datasets for data mining tasks: are they diverse/realistic/challenging?
- Impact of data quality (redundancy, errors, noise, bias, imbalance, ...) on qualitative evaluation
- Propagation/amplification of data quality issues on the data mining results (also interplay between data and algorithms)
- Evaluation of unsupervised data mining (dilemma between novelty and validity)
- Evaluation measures
- (Automatic) data quality evaluation tools: What are the aspects one should check before starting to apply algorithms to given data?
- Issues around runtime evaluation (algorithm vs. implementation, dependency on hardware, algorithm parameters, dataset characteristics)
- Design guidelines for crowd-sourced evaluations
The workshop will feature a mix of invited speakers, a number of accepted presentations with ample time for questions since those contributions will be less technical, and more philosophical in nature, and a panel discussion on the current state, and the areas that most urgently need improvement, as well as recommendation to achieve those improvements. Workshop submissions will be published in the CEUR-WS workshop series. An important objective of this workshop is a document synthesizing these discussions that we intend to publish at a more prominent venue.
*Submission*
Papers should be submitted as PDF, using the Springer LNCS style, available at https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines. Submissions should be limited to ten pages and submitted via Easychair at https://easychair.org/conferences/?conf=edml20.
Papers will be reviewed by at least two members of the Program Committee on the basis of technical quality, relevance, significance, and clarity. Submitting a paper to the workshop means that if the paper is accepted at least one author should present the paper at the workshop. Accepted papers will be published after the workshop with CEUR-WS or Springer.
*Important dates*
Submission deadline: June 09, 2020
Notification deadline: July 07, 2020
SDM early bird registration deadline: July 20, 2020
Camera ready: July 21, 2020
Conference dates: September 14-18, 2020
*Organizers*
Eirini Ntoutsi, Leibniz University Hannover & L3S Research Center, Germany, ntoutsi@kbs.uni-hannover.de
Erich Schubert, Technical University Dortmund, Germany, erich.schubert@cs.tu-dortmund.de
Arthur Zimek, University of Southern Denmark, zimek@imada.sdu.dk
Albrecht Zimmermann, University Caen Normandy, France, albrecht.zimmermann@unicaen.fr
The workshop's website can be found at https://imada.sdu.dk/Research/EDML/2020/
A vital part of proposing new machine learning and data mining approaches is evaluating them empirically to allow an assessment of their capabilities. Numerous choices go into setting up such experiments: how to choose the data, how to preprocess them (or not), potential problems associated with the selection of datasets, what other techniques to compare to (if any), what metrics to evaluate, etc. and last but not least how to present and interpret the results. Learning how to make those choices on-the-job, often by copying the evaluation protocols used in the existing literature, can easily lead to the development of problematic habits. Numerous, albeit scattered, publications have called attention to those questions and have occasionally called into question published results, or the usability of published methods. At a time of intense discussions about a reproducibility crisis in natural, social, and life sciences, and conferences such as SIGMOD, KDD, and ECML/PKDD encouraging researchers to make their work as reproducible as possible, we therefore feel that it is important to bring researchers together, and discuss those issues on a fundamental level.
A related issue is directly related to the first choice mentioned above: even the best-designed experiment carries only limited information if the underlying data are lacking. We therefore also want to discuss questions related to the availability of data, whether they are reliable, diverse, and whether they correspond realistic and/or challenging problem settings.
*Topics*
In this workshop, we mainly solicit contributions that discuss those questions on a fundamental level, take stock of the state-of-the-art, offer theoretical arguments, or take well-argued positions, as well as actual evaluation papers that offer new insights, e.g. question published results, or shine the spotlight on the characteristics of existing benchmark data sets.
As such, topics include, but are not limited to:
- Benchmark datasets for data mining tasks: are they diverse/realistic/challenging?
- Impact of data quality (redundancy, errors, noise, bias, imbalance, ...) on qualitative evaluation
- Propagation/amplification of data quality issues on the data mining results (also interplay between data and algorithms)
- Evaluation of unsupervised data mining (dilemma between novelty and validity)
- Evaluation measures
- (Automatic) data quality evaluation tools: What are the aspects one should check before starting to apply algorithms to given data?
- Issues around runtime evaluation (algorithm vs. implementation, dependency on hardware, algorithm parameters, dataset characteristics)
- Design guidelines for crowd-sourced evaluations
The workshop will feature a mix of invited speakers, a number of accepted presentations with ample time for questions since those contributions will be less technical, and more philosophical in nature, and a panel discussion on the current state, and the areas that most urgently need improvement, as well as recommendation to achieve those improvements. Workshop submissions will be published in the CEUR-WS workshop series. An important objective of this workshop is a document synthesizing these discussions that we intend to publish at a more prominent venue.
*Submission*
Papers should be submitted as PDF, using the Springer LNCS style, available at https://www.springer.com/gp/computer-science/lncs/conference-proceedings-guidelines. Submissions should be limited to ten pages and submitted via Easychair at https://easychair.org/conferences/?conf=edml20.
Papers will be reviewed by at least two members of the Program Committee on the basis of technical quality, relevance, significance, and clarity. Submitting a paper to the workshop means that if the paper is accepted at least one author should present the paper at the workshop. Accepted papers will be published after the workshop with CEUR-WS or Springer.
*Important dates*
Submission deadline: June 09, 2020
Notification deadline: July 07, 2020
SDM early bird registration deadline: July 20, 2020
Camera ready: July 21, 2020
Conference dates: September 14-18, 2020
*Organizers*
Eirini Ntoutsi, Leibniz University Hannover & L3S Research Center, Germany, ntoutsi@kbs.uni-hannover.de
Erich Schubert, Technical University Dortmund, Germany, erich.schubert@cs.tu-dortmund.de
Arthur Zimek, University of Southern Denmark, zimek@imada.sdu.dk
Albrecht Zimmermann, University Caen Normandy, France, albrecht.zimmermann@unicaen.fr
The workshop's website can be found at https://imada.sdu.dk/Research/EDML/2020/
Journal Promotion
|
| Acta Koreana (Acta Koreana) |
|
| Acta Via Serica (AVS) |
|
| Computers and Concrete (CAC) |
|
| Journal of Information Processing Systems (JIPS) |
|
| Journal of Internet Computing and Services (KSII - JICS) |
|
| Journal of Smart Tourism (Smart Tourism) |
|
| Membrane and Water Treatment (MWT) |
|
| Wind and Structures, An International Journal (Wind and Structures) |
MANUSCRIPTLINK © 2015 -
