Data correction 21.06.24
We reviewed and corrected several structures following remarks of challenge participants.
The corrections were done mainly for mixtures and salts.
The structure check was extended to prioritise structures as provided by ACS CAS service.
In cases where CAS suggestions were ambiguous, we used structures as retrieved PubChem. The structural information and descriptors were updated.
Leaderboard set was released 15.08.24
Can I use commercial software?
Yes, on the condition that access to this software (including, if required, free licenses
to use it) and instructions to reproduce the model are provided during the validation
step.
See more details here.
Can I use use additional data?
Yes, as long as they are not from the same assay (in terms of type and target) as
the one used by the EPA in their Transthyretin (TTR) study.
Can I use use Tox21 data? Yes, you can.
Why are you planning to release the Leaderboard set?
This set will be released on August 15th. When doing multiple submissions one can
partially or even fully identify activity values for all molecules in this set. Thus,
it is better to release this set publicly: that way there is no need to reverse engineer
it and all participants will be under the same conditions. The participants can (and
should) use it to improve their models for the final set, which will stay completely
blind during the whole competition. Once the leaderboard set is released, results
for this set will no longer serve as a yardstick for the predictive power of models.
Thus, after the public release of the leaderboard set each participant can in theory
submit just one prediction, since there will be no feedback on how good the new model
is compared to the estimation each participant makes themselves.
Are we allowed to use docking scores obtained with, e.g. AutoDock Vina, as descriptors?
Yes, definitely.
My descriptor calculation program fails for some compounds in the test set. What should
I do?
We had a similar issue for ToxCast EPA challenge. For such compounds we used an average value of all training set molecules as predicted
values. Definitely a better strategy should also exist :) and you are welcome to find
it.
Are multiple submissions allowed in the competition?
Each participant can join only one team. The team can submit as many predictions as
required. The last submission (submitted before September 1st) will be used to score
the blind set. Please make sure that the last submission is the best one.
Why does the Leaderboard show results submitted before and after August 15th separately?
Since the leaderboard set is now available, participants could submit the experimental
values as predictions and receive RMSE = 0. That is why only results submitted before
the August 15 should be considered as true predictions. Of course, some participants
may have provided correct prediction results for this set even after August 15th.
The results for the blind set, which will be used to score submitted models, will
be published after September 1st.
What is the best strategy for estimating model performance for the blind set?
Use cross-validation (CV) or bagging to determine the performance of your model (also
include the leaderboard set as part of the training set) and use the CV results for
the leaderboard set when submitting your predictions. Or use the proposed training/leaderboard
set split. This should provide you with a reliable estimate of the model performance
for the blind set. Rebuild the model with all data using the same protocol as applied
during the CV. If you apply a model built with all data (including the leaderboard)
to the leaderboard set, you will see overfitted results (compare "itetko_acs"" models
before and after 15.08). The true performance of this and other models will be tested
on the blind set.
Which activity is measured against Transthyretin (TTR)? Can you please provide a
description of the activity measure used? Can you comment on the compound concentration(s)
used with respect to the activity measure?
Read: Eytcheson, S. A.; Zosel, A. D.; Olker, J. H.; Hornung, M. W.; Degitz, S. J.
Screening the ToxCast Chemical Libraries for Binding to Transthyretin Chem. Res. Tox. 2024, 37, 10, 1670-1681, also mentioned at https://ochem.eu/static/challenge.do
After using the training and leaderboard sets for training the model, should the
predictions for the entire test set be submitted or only the blind set?
Yes, predictions should be submitted for the full test set of 500 compounds. Note
that you are not required to use the leaderboard set to develop your model. It was
released just to provide equal conditions to all participants.
Given that the leaderboard set is already released, can I still submit multiple predictions
to see the calculated RMSE? Would this allow for considering my submission per email
as the final prediction (I won't submit any predictions after sending you the email)?
You can submit predictions as many times as you wish. If you would like to estimate
accuracy of your model on the leaderboard set, do not include this set in the training
set of your model. Please, do not send any predictions by e-mail if you have an account
at OCHEM. Submissions by e-mail is only for the (few) participants who do not have
an account on OCHEM. In case of mistakes, in format, etc., there will be no possibility
to correct them. We will upload predictions from the email to OCHEM AFTER release
of data for blind set. This is also additional work for the organisers. Therefore,
everybody is encouraged to create an account and submit predictions on-line.
Can I use external descriptors in OCHEM?
Yes, you can upload them for all molecules from the training+test set as described
at Descriptor Storage.
They will appear on the descriptor page. Once you have developed a model with them,
save the model.
After this you can create a respective entry at Comprehensive Modelling and use as any other set of descriptors to develop multiple models.
How can I upload external descriptors?
Contact us challenge@icann2024.org to allow using this feature (which was disabled by deafault). Follow instructions
as shown at
example how to upload descriptors Contact us in case if somehing do not work.
You ask to predict 500 molecules but 200 of them are the leaderboard set. Will submission
of the overfitted results for leaderboard set improve model score for test set?
No. Accuracy of models will be tested on 300 blind set molecules, which will not be
available to challenge participants before the end of challenge.
I still have not registered for the Challenge. I understood that some validation
will be required. Is it too late to join the Challenge now?
Validation will be done only for teams with top performances after the Challenge ends.
You can open an account at OCHEM and submit your predictions (or send them by e-mail,
note that it is your responsibility to provide them in a correct format) even a few
minutes before the deadline 31 August 23:59 (UTC+14:00) to become an eligible participant of the Challenge.
Final predictions upload.
If you have an OCHEM account: any email submission will be ignored. Just upload your
final results and that will be enough.
Any email asking to sort data and predictions and to deal with erroneous predictions
will also be ignored. The option to send results by email was only for participants
who for some reason could not register.
This option was a mistake (only one team sent results this way so far but >10 participants
with OCHEM account sent their predictions by e-mail) and will not be available in
the future.
If you have other questions:
Send e-mail to challenge@icann2024.org. Answers will be also posted here.