Loss landscape

The first talk on the conference was really a good one. Stanislav Komanec presented interesting research about loss landscapes. We actually think that they are very rugged, but he showed, that it is possible to construct a hyperplane with only a fraction of dimensions which intersects the basin where the optimum is lying. Also, given any point and a minimum, when those two are connected by a linear path and we plot the loss on this path it actually looks like reversed sigmoid function, although we are used to think that there are many hills in the loss landscape.

He followed with sampling points around the minima and then talking about ensembling. Combining predictions of multiple ML models usually leads to better generalization and accuracy. This equals to combining predictions from different minima basins. Ideally, using those hyperplanes that intersects basins, we should be able to traverse from one hyperplane to another, from one basin to another and then ensemble. However, in practice the intersections are too thin and we also don't know which basins actually generalize well.

Machine learning for music

I think that this presentation was my favourite one, although I am not sure that I have learnt something new ML-wise. It was definitely the best prepared and presented presentation. It was basically intro into how sounds can be processed using ML, i.e. converting sounds to spectrograms and then checking how the sounds looks like in spectrograms, e.g. that percussion looks like very bright horizontal but is very thin, because the sound does not last look, but is on many frequencies, while more harmonical sounds are horizontal instead of vertical.

Interesting resouces from Lenka Zdeborová

AI Act

XY presented very interesting presentation on the AI Act which is a collection of regulations regarding data and machine learning. I actually wasn't expecting that we will hear a presentation purely about a law, but it was actually one of the best presentations there. It was great to hear about it in advance because I think it will get to the media only when it is actually passed and by then people will think about it in the same way as about GDPR - no-one actually knew what is needed and that it is actually very helpful law. I think it will be similar with the AI Act.

The AI Act composes of multiple smaller laws, GDPR is actually a part of it. I don't remember the names of individual laws but following should be included:

  • ML applications will be divided in categories according to the risk they pose to the society. There will be a category of ML applications with unacceptable risk and all companies in EU will be prohibited from using ML for specific purposes like social ranking. The important point here is that the division is by intent not by algorithm. It is definitely good to know that AI won't be used in EU the same way as in China.
  • Another point that will be required by one of the laws is that companies that use ML will have to allow users non-profiling, i.e. an option to opt out from ML-based recommendations. I am looking forward to when Spotify will show me different songs than those which are currently in my bubble.
  • It will be also necessary, if a company is using AI, to not only store all the input data but also the model that predicted and its predictions.

Enhancing drug discovery with NLP

In this project they went through 30M+ abstracts of papers on PubMed, generated embeddings and tried to classify already know molecules, in order to figure out, whether drugs have additional unknown effects or not as lately it was found out that there is a cancer drug, which actually has an antibiotics effect as well.

They were looking at proteins and found out that it is possible to learn from these text embeddings of PubMed abstracts and that the prediction is also greatly enhanced by generating text embeddings from the protein triplets sequence. They also reported that patent texts actually did not help the predictions probably because they are written in such a way not to disclose too much information to the competition.

Some additional notes from other talks

  • Big companies use Triton from NVIDIA in production to publish their GPU models.
  • Rossum is a company that works on automatic text extraction from documents. They don't have one ultimate model for reading documents, instead they broke the task of reading into smaller parts like - classification whether the text is text, abstract, figure, table, etc., extraction data from table, image to text conversion, etc.
  • ONNX from Private.AI is used to speed up PyTorch models in production.
  • Interesting presentation about drift in ML models.