{"id":25302,"date":"2021-03-09T14:07:58","date_gmt":"2021-03-09T13:07:58","guid":{"rendered":"https:\/\/lium.univ-lemans.fr\/?p=25302"},"modified":"2021-03-11T09:11:56","modified_gmt":"2021-03-11T08:11:56","slug":"minibert-a-simple-and-explainable-bert-model","status":"publish","type":"post","link":"https:\/\/lium.univ-lemans.fr\/en\/minibert-a-simple-and-explainable-bert-model\/","title":{"rendered":"MiniBERT: a simple and explainable BERT model"},"content":{"rendered":"

Seminar from Ga\u00ebtan Caillaut, Post-Doctoral researcher at LIUM <\/h2>\n

 <\/p>\n

Date:<\/strong> 12\/03\/2021
\nTime:<\/strong> 11h00
\nLocalization:<\/strong> online<\/a>
\nSpeaker: <\/strong>Ga\u00ebtan Caillaut<\/a><\/p>\n

 <\/p>\n

MiniBERT: a simple and explainable BERT model<\/strong><\/p>\n

 <\/p>\n

As part of the PolysEmY project<\/a>, we work with the SNCF (French railway company) to produce “polysemic-aware” word embeddings. Documents provided by the SNCF are written in technical vocabulary, specific to the SNCF. It is hence difficult to re-use models trained on generalist corpora (such as Wikipedia) since they cannot take into account the specificities of the SNCF\u2019s documents. Especially, a lot of acronyms are used, and many of them (more than 40%) are polysemous.<\/p>\n

We have two main goals, which are (1) capturing polysemous information from text while (2)keeping the model simple and explainable. We assume that a word meaning can be deduced from its context, this is why we think the attention mechanism is perfectly suitable to encode polysemous words, since it allows to weight each pair of words according to their relative relevance according to a given criterion (here, the criterion is the semantic influence of one word on another).
\nWe also try to keep our model as simple as possible, since simple models are naturally easier to explain and understand than models compounded of billions parameters (such as BERT or GPT-3). Furthermore, since we work on a relatively small corpus, and because we focus on a single task (capturing polysemy), we think that a model as powerful as BERT\u202fis not required.<\/p>\n

During my presentation, I will introduce the PolysEmY project on which I\u2019m working. Then I\u2019ll introduce the MiniBERT model and our motivations to work on this extreme simplification of the BERT model. You will also see that, while being quite simplistic, MiniBERT\u2019s performances are actually competitive and its output are easily explainable.<\/p><\/div><\/div><\/div><\/div>

 <\/div>

\"\"<\/p><\/div><\/div>

 <\/div><\/div><\/div>","protected":false},"excerpt":{"rendered":"

Seminar from Ga\u00ebtan Caillaut, Post-Doctoral researcher at LIUM   Date: 12\/03\/2021 Time: 11h00 Localization: online Speaker: Ga\u00ebtan Caillaut   MiniBERT: a simple and explainable BERT model   As part of the PolysEmY project, we work with the SNCF (French railway company) to produce “polysemic-aware” word embeddings. Documents provided by the SNCF are written in technical […]<\/p>\n

READ MORE<\/span><\/i><\/span><\/a><\/p>\n","protected":false},"author":14,"featured_media":13238,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[43],"tags":[49],"acf":[],"_links":{"self":[{"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/posts\/25302"}],"collection":[{"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/comments?post=25302"}],"version-history":[{"count":0,"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/posts\/25302\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/media\/13238"}],"wp:attachment":[{"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/media?parent=25302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/categories?post=25302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lium.univ-lemans.fr\/en\/wp-json\/wp\/v2\/tags?post=25302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}